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INTRODUCTION 

This evaluation toolkit was written in 2004 to help voluntary and community 
arts organisations in Northern Ireland to evaluate their work, especially their 
social impact on participants. Arts organisations in Northern Ireland are 
subject to strong pressures to measure their achievements and some already 
have excellent evaluation systems. This toolkit aims to increase the 
consistency of evaluation work so that individual arts organisations can better 
understand and explain their effects, but also so that the entire sector can 
make a stronger case to the Government. The toolkit is the first stage in a 
larger process that will a) see evaluation extended to all sectors of the arts, 
and b) will evaluate the impact of arts organisations' work on audiences as 
well as participants. 

This toolkit was written in part in response to demand from arts 

organisations in Northern Ireland. Annabel Jackson Associates carried out a 
survey of arts organisations in 2003 which found that: 

■ 90 per cent of respondents thought that more or better evaluation 
information would help their organisation. 

■ 79 per cent of arts organisations believed that evaluation would improve 
the effectiveness of the organisation. Motivating staff and volunteers, and 
helping with fund-raising were secondary objectives (56 per cent and 49 
per cent respectively). 

■ Arts organisations were keen to improve their own skills in evaluation. 
Some 70 per cent thought that the best structure was to support arts 
organisations in self -evaluation, with only 21 per cent thinking that the Arts 
Council should take responsibility for evaluation. 

The toolkit was written by Annabel Jackson working in a participative way 
with an evaluation steering group. Members of the evaluation steering group 
commented on drafts of the toolkit and tested the ideas in their organisations. 

The members of the evaluation steering group were: 

■ Andersonstown Traditional and Contemporary Music School 

■ Arts & Disability Forum 

■ Belfast Community Circus 

■ Community Arts Forum 

■ Kids in Control. 

■ Mid-Armagh Community Network 

■ Open Arts 

■ The Armagh Rhymers 

■ The Playhouse 

■ The Verbal Arts Centre 

■ Wheelworks 
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We would like to express our warm appreciation for the time, energy and 
enthusiasm of our excellent steering group. We hope that the product of our 
effort is as useful to apply as it was enjoyable to create. 

Annabel Jackson Associates also produced a review of evaluation toolkits 
and performance measurement systems in the arts, which is available on 
request. 

The toolkit was designed according to the following principles: 

■ The toolkit should ask questions of interest to arts organisations and 
participants. These should be questions the answers to which have the 
capacity to change lives and elevate communities. 

■ The toolkit should make evaluation as easy as possible. The text should 
be action oriented. Explanation should be relegated to an optional 
reference section. 

■ The toolkit's evaluation framework should compress the largest amount of 
meaning into the smallest amount of work. It is essential to respect the 
time of staff, volunteers and participants. 

■ The toolkit should respect the variation in voluntary and community 
organisations. It should be flexible enough to suit arts organisations of 
different sizes, art forms, cultures, functions and stages in their 
development. 

■ The toolkit should operate at different levels. We have defined core 
indicators that are the minimum necessary to demonstrate the value of the 
arts sector. We also provide guidance to help arts organisations that want 
to do more to evaluate the full range of their social impacts. 

■ The toolkit should be mixed method: quantitative and qualitative. The 
method should be chosen according to the question being pursued, not 
according to a pre-existing ideology. 

■ The toolkit should respect the values of the voluntary and community arts. 
This sector adopts an open, developmental structure that gives importance 
to the process as well as the product. 

The evaluation toolkit is aimed at helping arts organisations to increase their 
own capacity for evaluation. Self evaluation is helpful because it puts you in 
control. You know your organisation best. Self evaluation puts you ahead of 
your funders' demands rather than on the back foot. 

The evaluation toolkit shows you how to integrate evaluation into your day to 
day operations. Linking evaluation to management systems minimises the 
amount of work required. It is easier to collect information as you go along 
rather than leaving all the paperwork to do at a later stage. Reporting is more 
useful because you get feedback live rather than after the event. This process 
of integration is as relevant to an organisation with one employee as one with 
five or ten employees. 

Evaluation links to other information systems in your organisation. You will 
still need to monitor your financial performance, your relationship with 
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stakeholders such as funders and members, and any staff that you have. The 
skills that you learn in evaluation will help you with these tasks. 

Evaluation can form the foundation for your organisation's business plan. 
The evaluation tools explained in this toolkit will help you to clarify your 
objectives, assess your organisation's strengths and weaknesses, and set 
your priorities. 

There is no one way of doing evaluation. You should be able to find an 
approach that matches your culture and resources. The toolkit is designed to 
give you the skills to customise an evaluation approach to your own needs 
and circumstances. 

Annabel Jackson 

April 2004 
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Stages in evaluation 

There are three stages to evaluation: before you start your project/activity, 
during your project/activity and after your project/activity 

The rest of this toolkit uses the word "project" to apply generally to the work 
(whether self contained, like a project, or continuous, like your core activity) of 
a voluntary or community arts organisation. 




During the 
project 
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Aspects of evaluation 

Each evaluation stage requires you to ask yourself questions, set up or apply 
systems, involve people and take action. The toolkit uses these headings: 




Framing, focusing, conceptualising, interpreting, synthesising 



Planning, gathering data, entering data on a computer, analysing data, 
disseminating information 



Motivating, training, involving, reassuring, listening, informing, influencing 



Making recommendations, implementing recommendations 



All four elements are important. If you neglect thinking, you might be 
missing out important issues . If you neglect systems, you might be collecting 
weak or incomplete information. If you neglect people, evaluation might 
generate resistance rather than interest. If you neglect action, valuable 
insights from evaluation might be lost. 

Most organisations are naturally better at one of these four. One of the 
purposes of this toolkit is to give you practical support so that you can do 
each satisfactorily. 
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BACKGROUND ON EVALUATION 

What is evaluation? 



Evaluation is the art of asking 
interesting and provocative questions. 
Evaluation provides guidance to help 
you to frame precise, unloaded 
questions. Evaluation can help you to 
see through the mass of mundane or 
"interesting but not very useful" 
questions to find the true, fundamental 
questions at the heart of your work. 



Evaluation is about evidence. Evaluation 
draws on a whole host of research 
methodologies that can give you a different 
way of seeing your work. Quantitative 
information (data) can reveal patterns that are 
not obvious, for example where particular 
groups of participants have benefited 
significantly more than the average. Evaluation 
is not a replacement for intuition. However, it 

can help you to test and explain your intuitions. 



Evaluation is about causation. Evaluation is 
at its best not when it is describing or 
summarising your work, but when it is 
investigating what is achieved, why, how and 
when. Evaluation can help to explain the 
constraints and contextual factors that affect 
your work and therefore to help to ensure 
expectations (yours and your funders') are 
realistic. 



Evaluation is about different perspectives. 
Even the simplest project or activity can 
benefit from asking different stakeholders 
(participants, artists, partners and funders) 
about their experiences and perceptions of 
your work. 




Evaluation is about reflection. Evaluation can 
provide a structure to prompt and record 



feedback on your achievements. You probably 
already have observations about what works 
and what doesn't work in projects or activities. 
Evaluation helps you to make the best of these 
observations so that they are not lost in the 
busy day-to-day life of your organisation. 





Evaluation is about learning. Evaluation is a 
continual process of questioning, seeking 



evidence and reflecting on the findings. 
Through time this will allow you to improve 
your work and ensure that you make the best 
use of the resources that you have available. 
Thus evaluation not only measures, but also 
contributes to success. 
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Benefits of evaluation 



Evaluation can 
make your 
organisation more 
confident in telling 
funders what works 
and what doesn't. 




Evaluation can help you 
to motivate staff and 
participants. 
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Myths about evaluation 



Evaluation is about proving 
the value of your work. On 
the contrary: evaluation is 
more interesting and powerful 
if it asks broader questions 
about what your work 
achieves and how. Evaluation 
information can feed into 
advocacy but evaluation 
questions should be neutral 
rather than tainted by 
advocacy. 



Evaluation is only for 
funders. On the contrary: 
funders tend to have fairly 
broad requirements of 
evaluation. The most 
meaningful and useful 
evaluation system will be that 
designed to investigate your 
own questions. Information for 
funders should be generated 
as a side product from this. 



Evaluation is all 
numbers. On the 

contrary: most evaluation 
mixes qualitative and 
quantitative approaches. 



Evaluation is 
retrospective. On the 

contrary: evaluation is 
more useful if it runs 
alongside your work. This 
means that early 
conclusions can provide 
feedback to allow you to 
fine tune your work. 




Evaluation is negative. On the 

contrary: evaluation can 
celebrate achievements as 
well as highlighting gaps 
between expectations and 
reality. 
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SUMMARY 




Ensure you have a clear idea of what your project is intended to achieve. In 
particular, you need to think about: 

■ What are the resources of the project? Include human, financial, 
organisational and community resources. 

■ What are the activities of the project? Include processes, events, 
technology and actions that are part of the project. This includes the number 
and type of outreach events or other publicity, the number and type of 
sessions and the number and type of performances or exhibitions. Be aware 
of the choices that you have made at each stage in the project and, by 
implication, of the alternatives that are possible. 

■ What are the intended outputs? Generally this is the number and type of 
participants in an arts project and the number of visitors or audience 
members. However, it is relatively common to see the term "outputs" used to 
describe what we have listed here as activities as well as outputs . 

■ What are the intended outcomes? This refers to changes in the 
behaviour, skills or attitudes of participants. 

■ What are the intended impacts? Impacts are community-wide benefits 
whether in social or economic terms. 

Arts organisations don't tend to have tightly prescribed intended outcomes 
like other programmes that are explicitly designed to avoid teenage 
pregnancy or reduce crime. However, arts projects still have broad 
outcomes, such as helping participants to increase in confidence and helping 
them to express themselves. 



There are nine questionnaires that must be completed (where relevant) by all 
organisations receiving core funding (RFOs) or project funding over £10,000: 

■ Organisational Form. This gives background information and is relevant 
to all organisations. Only one form needs to be submitted and thereafter 
updated annually. 

■ Support Services Form. This is relevant if your organisation provides any 
support services to artists or arts organisations. 

■ Financial Form. This is relevant to all organisations. 

■ Attendance Sheet for Individuals. This is relevant for all projects where 
90% of the group are aged 1 7 or over. 

■ Activity Form for Participation. This form will be relevant to most 
voluntary and community arts organisations. It gives information on any 
educational workshops, classes, or other participative projects carried out. 
One form is filled in per project. 

■ Activity Form for Productions/Concerts. This short questionnaire gives 
information on productions, so that this does not need to be repeated for 
each performance listing. 

■ Activity Form for Performances/Concerts. This will be cover most of the 
work of performing arts organisations as well as the larger live performances, 
live shows and concerts produced by the voluntary and community arts. 

■ Activity form for Exhibitions. This covers the work of galleries as well as 
larger exhibitions from voluntary and community arts organisations. 

■ Activity Form for Publications. This refers to any catalogues, 
newsletters, books or periodicals for the arts sector last year. Do not include 
marketing brochures or publicity leaflets. 

The Activity Form for Participation includes a section on exhibitions and 
concerts that are an inherent part of the participative process. Use your 
judgement to decide whether performances or exhibitions are of sufficient 
scale to warrant separate listing in the Activity Form for 
Performances/Concerts or the Activity Form for Exhibitions. 
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Copies of all obligatory questionnaire are given in Appendix Two. There is a 
simple spreadsheet for recording answers, to be analysed by ACNI. 

There are eight optional questionnaires that you could use in your 
organisation: 

■ Enrolment Form. This records basic information about clients on 
participative projects. Information is collected at the first or second contact. 

■ Attendance Sheet for Groups. This records attendance for each session 
of a participative project. 

■ Quality Form and Observation Form. These are optional forms that you 
can use to help to record internal reflections on your work. 

■ Outcomes Questionnaire. This measures the short term impact of your 
project on the participants. 

■ Outcomes Questionnaire for Young People. This has been designed for 
use by people aged 11-16. 

■ Artist's Evaluation Form, Teacher's/Youth Leader's Evaluation Form 
and Organisational Head's Evaluation Form. These are optional forms for 
youth arts projects. They can be completed by a sample or all people 
involved in the project, in writing or over the telephone. 

■ In addition, ACNI will be carrying out a survey using the Follow up 
Questionnaire. This provides more detailed information on long term the 
impact of your project on participants. 

Drafts for all the optional questionnaires are given in Appendix Three. 





Action 



You need to ensure that evaluation is taken seriously in your organisation. 
This will greatly affect the quality of the data that you receive and therefore 
the value of the process. Consider whether you need to: 

Train staff, freelancers and committee members in evaluation. 

Have internal feedback sessions, or an agenda item for evaluation in 
existing staff meetings . 

Recruit an evaluator for specific tasks or on a retainer. 

Include time to complete feedback forms in the timetable for events. 

Put evaluation or performance measurement on the agenda for your 
management committee. 

Put evaluation in the job descriptions for staff or contractors. 

Include a budget for evaluation in project proposals or funding applications . 

Purchase additional software for data analysis . 

Allocate a specific file (in your computer and in your filing cabinet) for 
feedback forms and other evaluation data. 

Think about how to ensure that evaluation data is kept confidential. 

Talk to other arts organisations about how they do evaluation . 

Have a wall chart showing recent achievements. 

Whatever your internal arrangements, you will need to tell participants and 
staff early on that the project will be evaluated. You will also need to obtain 
permission from participants early on to pass on their details to ACNI if you 
wish to take part in the follow up survey. 

It is important to translate the finding of your evaluation into 
recommendations about practical ways to improve your project. You also 
need to allow time and perhaps resources to put these recommendations 
into action. You can't improve all your work simultaneously. It is sensible to 
focus your effort. For example, you could identify two key improvements to 
make if you re-run your project. 
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BEFORE STARTING THE PROJECT 

Plan the evaluation 



Planning the evaluation can enable you to make the best of the skills and 
resources in your organisation. Good planning will increase the quality of the 
data produced, smooth the workflow and ensure that results are useful. 

It is better to have one coherent set of questionnaires than to keep going 
back to participants to ask them additional questions that were missed out of 
earlier evaluations. Similarly, it is better to have accurate information on a 
small number of key questions than to have incomplete or biased information 
on a large number of questions. 

Evaluation should be proportional to the scale of work. Don't expect to 
measure change from a single workshop. A common guideline is that you 
should dedicate 1 per cent of the resources of a project to evaluation. 

A key element of the planning process is to consider how the evaluation 
proposed in this guide fits in with any existing evaluation that your 
organisation already carries out. 

There are two Thinking tasks: 
Draw the Project Logic 

A logic model is a picture of how your project works. It gives a visual 
presentation of the assumptions underlying your work, in particular the 
assumptions about how activities lead to outcomes. The logic model is a 
simple but very powerful tool that can help you clarify your thinking, measure 
progress along the way (through intermediate indicators) and communicate 
your objectives to different stakeholders. Logic models help funders and 
other stakeholders to understand the practicalities of projects and therefore 
to temper their expectations. 

The Kellogg Foundation, which developed and popularised logic models , 
explains them as follows: 

Your planned work Your intended results 




Certain 
resources are 
needed to run 
your project. 



If you have 
access to them, 
thenyou can 
use them to 
accomplish 
your planned 
activities. 



If you 

accomplish 
your planned 
activities to the 
extent you 
intended, then 
you hopefully 
will deliver the 
amount of 
service you 
intended. 



If you achieve 
your planned 
activities to the 
extent you 
planned, then 
your 

participants will 
benefit in 
certain ways. 



If these 
benefits to 
participants are 
achieved, then 
certain changes 
in organisations 
or communities 
might be 
expected. 



1 . Resources 



2. Activities 



| 3. Outputs 




For an example of a completed project logic model see page 21 . 

Logic models are read from left to right, as in this diagram . However, the best 
way to design them is to start with outcomes and work left. 
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To construct a logic model you need to ask: 

■ What are the intended impacts? Impacts are community-wide benefits 
whether in social or economic terms. 

■ What are the intended outcomes? This refers to changes in the 
behaviour, skills or attitudes of participants. 

■ What are the intended outputs? Generally this is the number and type of 
participants in an arts project and the number of visitors or audience 
members. However, it is relatively common to see "outputs" and "activities" 
described as outputs. 

■ What are the activities of the project? Include processes, tools, events, 
technology and actions that are part of the project. This would include the 
number and type of outreach events or other publicity, the number and type 
of sessions and the number and type of performances or exhibitions. 

■ What are the resources of the project? Include human, financial, 
organisational and community resources. 

Arts organisations don't tend to have tightly prescribed intended outcomes 
like other programmes that are explicitly designed to avoid teenage 
pregnancy or reduce crime. However, arts projects still have broad 
outcomes, such as helping participants to increase in confidence and helping 
them to express themselves. 

In order to draw your logic model you might want to ask yourself: 

■ Why was the project developed? What is the problem or need it is 
attempting to address? The rationale can be positive (increasing creativity) 
as well as negative (reducing isolation). 

■ Who are the stakeholders? Who are the people involved in the project? 
Are there people who would be interested in the project who are not 
involved? 

■ Who are the intended participants? Why were they chosen and to what 
extent has the project been tailored to their needs? 

• How will the projects achieve the intended benefits? What is it in the 
project that will benefit the participants? In evaluator's language this is asking 
about the mechanisms whereby outcomes are achieved. 

Don't make your logic model too detailed. It should fit on a page or it will be 
difficult to read. 

Test the Project Logic 

Once you have drawn the project logic you can consider whether you are on 
solid or weak ground, for example: 

■ Where is the initiative for the project coming from? If it is coming from 
funders then does it match your experience of what users need? 

■ Are the resources sufficient? Is the budget right? Are there elements 
missed out, like evaluation or training? Are you trying to do too much? Do 
you need other skills or resources? How could you obtain these? Do you 
need more partners? Are the time lines realistic? 

■ Is there a need for this project? Have users expressed a need for this 
type of project before? If not, do you need to do a survey to check need? If 
users have expressed a need for this type of project before, how many? Is 
this enough to support the project? You might want to investigate the terms 
on which potential participants want the new project: for example the 
preferred delivery conditions - most desire is conditional. There are two main 
methods for analysing need: written questionnaires or focus groups. You can 
also use other evidence of need such as waiting lists, responses to previous 
evaluation questionnaires and requests from other organisations. 

• Do different stakeholders have different expectations of your work? 

Are these compatible or do objectives need to be re-negotiated? Are 
stakeholders clear about what you are doing? How important is your work to 
their objectives? 
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■ Are the activities likely to lead to the outcomes? Are there any barriers 
that could prevent your target participants from using or benefiting from your 
proposed project? What could you do to remove barriers? This could include 
preparatory work or publicity to ensure take up. What could you do to 
increase the benefits? 

• Are there external factors that could affect your ability to reach your 
outputs or outcomes? You might like to make a list of the main external 
factors or risks that could affect your project and think about whether you can 
control or monitor them. 

■ Is the project right for your organisation? Does the project match your 
organisation's skills and resources? Does the project match your picture of 
where you want the organisation to be in the future? Is there synergy with 
other projects, either from your organisation or another organisation? Could 
you capitalise on this more? 

• How does this project fit with other organisations' work? Are other 
organisations already doing similar or complementary projects? Do they 
have information on the needs and delivery preferences of participants that 
would help you target your project? Could they help with recruitment or 
delivery? 



There are two Systems tasks: 
Produce an Evaluation Framework 

Your logic model should help you to identify the issues you need to evaluate . 
For example: 

■ What resources have been deployed in each stage of the project? Are 

resources available at the right time and in the right form for the project? Are 
the resources constraining what is possible in the project? 

■ What activities have been carried out? What are the key choices that 
have been made in delivering the project? What are the strengths and 
weaknesses of each of these delivery options? Does experience suggest 
that these delivery options need to be reviewed? 

■ What outputs have been produced? What is the level of activity? What is 
the level of take up? What is the background of project participants? 

■ What outcomes have been achieved? Have participants changed their 
behaviour, knowledge, skills or attitudes? What percentage of participants 
has changed in each possible outcome area? To what extent can these 
outcomes be traced to the work of your project? How does (what are the 
mechanisms whereby) your project achieve these outcomes? How does the 
pattern of outcomes differ over the short, medium and long term? 

■ What impact has been achieved? Has the project had an economic or 
social impact? How did external factors reinforce or counter the impact of 
your project? How could the impact have been greater? 

An evaluation framework outlines what methodology you are proposing to 
adopt to address your evaluation issues . 
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A possible structure is: 



Structure for an evaluation framework 





Participants 


Staff/volunteers 


Evaluator or peers 


nesourcco 




KA o n o rt q m out 
IVIdlldyclTlclH 

accounts 

Time sheets 

Appraisal information 

Questions to artists 
ana otner project 
operators about the 
support they received 




Activities 


Questions on 

OUlCOlTlcb 

questionnaire 


Postal questionnaires 
or yroup discussion 
about the strengths 
and weaknesses of 
the project and key 
issues 


Quality monitoring or 

OUbcl VdllOil 


Outputs 


Questions on 
background included 
in outcomes survey 


Registration forms 
Output forms 
Session records 




Outcomes 


Telephone or 
telephone survey 


Postal questionnaires 
or group discussion 




Impact 


Included in any 
outcome survey 


Compile figures on 
staffing from the 
output questionnaire 


Independent sector 
evaluation 



Produce an Evaluation Plan 

The evaluation plan helps you to see the work associated with evaluation. It 
should show: 

■ Tasks. What are the tasks required for each evaluation method? Appendix 
One gives a list of the tasks involved in each of the main evaluation methods. 

■ Responsibilities. If you have staff, think about who will do the work. You 
should think whether they have the skills and qualities to carry out the tasks. 
Clerical tasks, such as entering data, require precision, IT skills and the 
ability to respect confidentiality. Interviewing is a highly skilled task that 
requires the individual to suspend their own views and listen to the 
interviewees. Observation might require additional expert knowledge in order 
to make quality judgements. 

■ Deadlines. Ensure that you have sufficient time so that tasks are not 
squeezed out or compromised. Identify any tasks that have a lead in time 
and ensure these are given priority. Deadlines need to be realistic. Relate 
these to the rest of your work. Check that the times of heightened activity 
don't clash with other pressures on the organisation. For example, you might 
like to prepare and print out the evaluation questionnaires before they are 
needed. You might want your evaluation report to be ready for a specific 
conference or event or committee meeting, in which case you should s et 
deadlines by working backwards from the required reporting date. 

■ Targets. Set rough targets for response rates and outcomes. This is a way 
of being precise about your expectations. Be realistic. Telephone surveys 
should aim for a 90 per cent response rate, but postal surveys sometimes 
only receive 20-30 per cent of the questionnaires sent out. Appendix One 
gives guidelines to increase the response rate for written questionnaires. 
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• Any expenditure. 

Ensure that your plan is realistic. For example, consider: 

■ How far do you want to go? It is better to do a modest amount of 
evaluation well than to take on too much and find that the data is incomplete 
or inaccurate. 

• What are your organisation's strengths and weaknesses in 
evaluation? You could think about this in terms of the four elements used in 
this toolkit: thinking, systems, people and action. 

• What are the barriers to the plan being implemented? What do you 
need to do to overcome them? 

Evaluation might seem difficult at first because you have to think about each 
stage. It will become much easier as it becomes part of the way you 
automatically go about your day-to-day work. 



Structure for an Evaluation Plan 



TASKS 


RESPONSIBILITY 


DEADLINES/TARGETS 


EXPENDITURE 


Write or copy evaluation 
forms. This might include 
piloting and amending 
forms 






Depending on the 
method: paper, 
stamps, envelopes, 
telephone costs 


Distribute and collect 
evaluation forms. Set the 
sample, introduce the 
forms, collect them back, 
answer questions about 
the forms 






Files to keep 
questionnaires 


Enter data on the 
computer 






Spreadsheet software 
and possibly expert 
help 


Analyse the data 






Spreadsheet software 
and possibly expert 
help 


Write up. Translate 
findings into 
recommendations 








Disseminate the findings. 
Provide feedback to 
funders and respondents 






Email, poster, 
newsletter, website, 
annual report 


Launch the findings 






Depending on the 
type of event: paper, 
venue, food 


Implement the 
recommendations 










There are two People tasks: 

Ensure Evaluation is Treated Seriously in Your Organisation 

Consider whether you need to: 

■ Train staff, freelancers and committee members in evaluation. 



■ Have internal feedback sessions, or an agenda item for evaluation in 
existing staff meetings . 

■ Recruit an evaluator for specific tasks or on a retainer. 

■ Include time to complete feedback forms in the timetable for events. 

■ Put evaluation or performance measurement on the agenda for your 
management committee. 
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■ Put evaluation in the job descriptions for staff or contractors. 

■ Include a budget for evaluation in project proposals or funding applications . 

■ Purchase additional software for data analysis . 

■ Allocate a specific file (in your computer and in your filing cabinet) for 
feedback forms and other evaluation data. 

■ Think about how to ensure that evaluation data is kept confidential. 

■ Talk to other arts organisations about how they undertake evaluation . 

■ Have a wall chart showing recent achievements. 
Involve Interviewees 

Most of your data will come from interviewing individuals. You will be asking 
people to allocate some time to evaluation, for example by completing 
questionnaires or taking part in are prepared for the evaluation. 

There are four ways in which you can involve interviewees: 

■ Tell them about the evaluation. It is good practice to tell organisations 
and individuals about the evaluation at the earliest possible moment. Ideally, 
you should mention at the beginning of the project that evaluation is an 
integral part of your work. Be sure to introduce evaluation in a positive light - 
for example, as part of your organisation's commitment to excellence and 
customer focus. The way you present evaluation affects the quality of the 
data that you receive. You really want to avoid the vicious cycle whereby 
evaluation is seen as a mindless bureaucratic exercise, which leads to 
respondents giving superficial answers consistent with a mindless 
bureaucratic exercise, and thereby reinforces their negative views of 
evaluation. 

■ Ask for their permission to use information. There are two levels of 
notification: 

o Organisations. Partners with which you are working, such 
as schools, might have formal procedures for approving 
surveys of staff or participants. You might need to apply for 
permission to compile data. You should also tell your 
partners when the evaluation will take place and you 
should offer them a copy of the final report. 

o Individuals. When they join the project participants (or 
parents/guardians for children) should sign Data Protection 
Act forms agreeing that information can be used for 
evaluation purposes. Participants should be told why the 
data are being collected and the use to which the results 
will be put. Most evaluators give and honour assurances 
that individual data will be confidential and that no personal 
repercussions will result from survey responses. You will 
need to remember this assurance when you are writing the 
evaluation report. 

• Minimise disruption to participants and participation. Think about the 
best stage to apply the questionnaires and allow time for this to happen. 

• Feedback conclusions and actions taken as a result. Keep people 
informed about how findings are used. If nothing ever changes as a result of 
evaluation then participants and staff won't want to continue filling in the 
forms. 

If they are told about the purpose of evaluation, consulted about the research 
methods and possibly involved in the research design, involved in respectful 
and non-threatening ways and given feedback about conclusions then you 
will find that most people are happy to be involved in evaluation. Asking 
stakeholders' views of a project is one way of showing that you value them. 
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There are two Action tasks: 
Monitor Your Evaluation Plan 

This has two components: 

■ Ensure work is completed on time. Tick off each task in your evaluation 
plan when it is completed. 

■ Ensure the quality of work is consistent. Check that interviewers are not 
suffering from burnout. This is noticeable from skipped questions, fewer 
probes or fewer open ended comments in interviews , and more transcription 
errors in data entry. Review the data every month to check that people are 
asking and recording questions consistently. 

Evaluate Your Evaluation 

At the end of each cycle of evaluation consider: 

• Whether you need to fine tune the evaluation methodology. You might 
need to reword questionnaires, give more explanation, remove questions 
that were not useful or include new issues that arise from the work. 

• Whether you need to change the way evaluation was carried out. 

Make notes on anything that needs to be changed in the future, for example, 
ensuring contact details are up to date so that interviewers do not have to 
spend time tracking participants. 

• What your organisation gained from evaluation. Be sure to keep notes 
of any benefits from evaluation so that you can use these as examples when 
you are motivating staff in the next cycle of evaluation. 



Simple form to evaluate your evaluation 





STRENGTHS 


WEAKNESSES 


Planning 






Choosing evaluation methods 






Compiling data 






Analysing data 






Writing up 






Communicating results 







Link to the Toolkit 



There are two Thinking tasks: 

Setting the Level of Detail for Evaluation 

Think about how much evaluation your organisation wants to do: whether 
you will: 

■ Use the minimum in the toolkit, the obligatory questionnaires. 

■ Use some or all of the additional questionnaires as well as the obligatory 
questionnaires. 

■ Add further subjects that are not in the toolkit. If this is the case, you might 
need further guidance on how to use methods and write questions (Appendix 
One). 




Action 
M M M 
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Your choice will reflect: 

■ The size of your organisation. 

■ The proportion of your work that is arts -related. 

■ The sophistication of your existing evaluation systems. 

■ The range of your work. Organisations that are continually developing new 
projects might want to do more evaluation than organisations with one 
consistent product. 

Note the Overlap with the Toolkit's Logic Model 

The logic model assumed in the toolkit is: 

Evaluation Toolkit's Logic Model 

Your planned work Your intended results 



Staffing 

Help in kind 

Money 

Premises 

Equipment 

Other 

resources 



Depth of 

engagement 

Accessibility 

Targeting 

Quality 

Leverage 



Workshops 




Enjoyment 




Organisational 


Outreach work 




Confidence 




impact 


Performances 




Motivation 




Social impact 


Number of 




Creativity 




Economic 


participants 




Self expression 




impact 


Profile of 




Interest in the 




participants 




arts 






Audience 




Participation in 






numbers 




the arts 
Skills 
Interest in 
education 
Social networks 
Tolerance 
Health 
Happiness 







1. Resources 



2. Activities 




4. Outcomes 




Outcomes will inevitably vary between projects and organisations. No project 
is expected to achieve all possible outcomes. Ideally, over time arts 
organisations and the Arts Council will grow to understand what kinds of 
projects generate which types of outcomes under which circumstances. This 
could produce different logic models for different types of projects. However, 
the logic models will always be speculative rather than predictive. Arts 
projects evolve organically during the artistic process. Arts projects are not 
production lines to produce fixed outcomes. 



There are two Systems tasks: 

Link to the Toolkit's Evaluation Framework 

We have drawn this toolkit as widely as is reasonable. However, you might 
have other evaluation requirements, especially from other funders. You need 
to combine these two sets of requirements so that you only have one set of 
evaluation questionnaires for each project. 

The approach that we are suggesting is probably a bit more quantitative than 
most arts organisations would come to naturally themselves. Please give 
quantitative methods a chance: a few numbers can save you a lot of words 
and will have greater credibility with funders. With a little care quantitative 
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methods can be as meaningful and sensitive as qualitative. For example, the 
proposed forms look at the depth as well as the breadth of participation. 

The evaluation methods that we propose are: 



Evaluation Toolkit's Evaluation Framework 





Questionnaires/Sources 


Resources 


Time sheets 
Appraisal information 
Organisational Form* 
Support Services Form* 
Financial Form* 


Activities 


Quality Form* 

Observation Form* 

Artist's Evaluation Form* 

Teacher's/Youth Leader's Evaluation Form* 

Organisational Head's Evaluation Form* 


Outputs 


Enrolment Form* 

Attendance Sheet* 

Activity Form for Participation* 

Activity Form for Productions/Concerts* 

Activity Form for Performances/Concerts* 

Activity Form for Exhibitions* 

Activity Form for Publications* 


Outcomes 


Outcomes Questionnare* 

Outcomes Questionnaire for Young People* 

Follow up Form* 

Artist's Evaluation Form* 

Teacher's/Youth Leader's Evaluation Form* 

Organisational Head's Evaluation Form* 


Impact 


Compile figures on staffing from the output questionnaire 
Compile figures on suppliers 
Independent sector evaluation 



* Draft questionnaires in Appendices Two and Three 



Clarify Overall Targets and Deadlines 

The evaluation toolkit will be rolled out gradually over the next year or so in 
this order: 

■ Members of the Evaluation Steering Group. 

■ Voluntary and community arts organisations receiving revenue funding. 

■ Voluntary and community arts organisations receiving project funding. 

■ The rest of the arts sector. 

The Organisational Form, Support Services Form, Financial Form and 
Activity Forms will be obligatory from April 2005. 

All other questionnaires are optional. The Follow up Questionnaire will be 
applied by an Arts Council researcher for all organisations with clients who 
are 17 and over and without severe learning difficulties . These organisations 
need to complete the Attendance Sheet for each of their projects so as to 
provide contact details for the Follow Up survey. 




There are two People tasks: 

Decide Whether to Use External Evaluators 

If you are intending to expand your evaluation beyond that in the draft 
questionnaires then you might need additional expert advice. Guidance on 
using external evaluators is given in Appendix One. 
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Action 



Plan Evaluation Training 

Four areas of training are important: 

■ Random sampling. Random sampling selects specific individuals for 
interview from an overall population of participants. Data collectors should 
work hard to contact the named individuals rather than giving up too soon 
and selecting another name on the list. 

■ Consistency. Data collectors must ask the questions in the same way over 
time and across organisations. 

■ An open mind. Data collectors must suspend their own judgments during 
the interviewing process. Even very experienced evaluators find that 
respondents frequently surprise them : that their opinions are not what was 
expected. Data collectors must allow the space for these new views to be 
expressed. 

■ Being positive. Data collectors must communicate to respondents that the 
evaluation is important, their response is valued and the interview is not an 
examination to which there is one right answer. 

There is one Action tasks: 
Management Committee Approval 

It would be helpful to seek and receive your Management Committee's 
approval and support for your work on the evaluation toolkit. 
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DURING THE PROJECT 

Record the activity 

There are two Thinking tasks: recording inputs and outputs 

Thinking " '"P 11 * 3 are staffin 9' m ° n ey and help in kind. 

Ill • Outputs are workshops, performances or exhibitions. 

Recording Inputs 

You will probably already have information on inputs from your internal 
management systems. 

Recording Outputs 

Keep track of activity as you go along. It is much more difficult and time- 
consuming to try and reconstruct a description of your work after the event. 
Record what happened, not what could have happened, might have 
happened, should have happened or will happen. 

This information is simple but very useful. It is surprisingly common for arts 
organisations not to know how many participants they have or had. This puts 
them at a great disadvantage in raising funds: if you have done the work, you 
should take the credit for it. 

The main point in measuring the outputs of voluntary and community sector 
arts organisations is to demonstrate the depth rather than the breadth of 
participation. You are unlikely to have large numbers of participants, but you 
could have intensive, detailed participation which you will want to 
communicate in your evaluation. This is not difficult to do. 



There are six obligatory forms: 
Attendance Sheet for Individuals 
Activity Form for Participation 
Activity Form for Productions/Concerts 
Activity Form for Performances/Concerts 
Activity Form for Exhibitions 
Activity Form for Publications 

Questionnaires are given in Appendix Two. These questionnaires will also 
give you the information to complete the Annual Survey of Regularly Funded 
Arts Organisations from the Arts Council (if relevant to you). 

There are two optional forms: 

Enrolment Form 

Attendance Sheet for Groups 

Questionnaires are given in Appendix Three. 

Getting Feedback on the Forms 

It is worth having a brief discussion in your organisation about the output 
evaluation. For example, as k project staff and volunteers : 

■ Do they feel involved in, and clear about the purposes and benefits of the 
evaluation? 

■ Are project organisers completing the forms consistently? If not, do you 
need to give further guidance on how to complete the forms? 
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■ Are the data accurate and timely? If not, do you need to give further 
explanation about the value of the data? 

■ Do the figures make sense? Are there other factors that need to be taken 
into account in interpreting the figures? 



There are two Action tasks: 



Interpret the Findings 

Look at the figures from the output evaluation and consider what they mean, 
for example: 



Were the inputs too low? 


Why? 


Did the project expand from its original 
conception? 

Did you underestimate the resources needed for 
the project? 

Did your resources change during the project? 


Were the inputs of a poor quality? 


Do you need to do more planning? 

Do you need to work with other organisations 

more? 

Do you need to raise money for future projects? 


Were the inputs too late? 


Were there problems from the terms of funding? 



Was the number of outputs lower 
than you expected? 


Why? 


Do you need to increase marketing? 
Do you need to strengthen partnerships? 


Were some target groups under- 
represented? 


Do you need to increase outreach? 

Do you need to fine tune publicity or delivery? 


Was the drop out rate high? 


Do you need to provide more information to 
potential participanb or partners for recruitment? 
Do you need to change the location, timing or 
other aspects of the delivery of the project? 
Do you need to provide incentives for 
participants to continue? 
Do you need to change the type of activity 
offered? 


Was the project over-subscribed? 


Do you need to expand the project? 


Was there a slow start in take up? 


Do you need stronger recruitment? 

Do you need a longer lead in time for setting up 

the project? 



Act on the Findings 

You need to consider whether the project is on schedule and on target. Low 
levels of participation will inevitably affect the outcomes achieved. This is one 
of the messages from the Logic Model. 

Monitor quality 

Think About Organisational Quality 

Thinking What does quality mean to your organisation? Think about things that you 

111 value, or aspects that you take for granted that would impress funders. This 

discussion will take more time if you have new staff or new projects. It will be 
more straightforward if you have a fairly standard product and a small 
consistent team. 
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A quality form is useful because: 

■ It helps to define who you are as an organisation (your unique selling 
point), what is different between what you do and what other organisations 
do. 

■ It shows that you care about quality. 

■ It allows you to make the best of any time you are observing a session. The 
form provides a few pointers, drawing your attention to aspects of quality, 
which helps when you are busy. 

■ It helps to depersonalise criticism. 
Possible aspects of quality could include: 

■ Was the project well organised? Was the project well publicised? Did 
everyone know what was going on? Did the different parts link together? Did 
the different parts occur at the right time? 

■ Was the project fit for purpose? Was the content appropriate to the 
participants ? Was the content at the right level of difficulty? 

■ Was the artistic quality high? See the next section. 
Think About Artistic Quality 

Comedia suggests that artistic quality has five components: 

■ Technique. Technical competence or sophistication. 

■ Originality. The newness of the content as well as the mode of 
expression. 

■ Ambition. The extent to which the art challenges creators and viewers to 
extend themselves beyond the norm. 

■ Connection. Relevance to the audience. 

■ Magic. Ability to provide non-rational responses, inexplicable and 
inexpressible reactions that stay with the audience. 

We are not suggesting that you measure these five in your outcome 
evaluation (i.e. by getting participants to rate your project on them). However, 
it is helpful to think about which ones are most important to you during any 
quality monitoring that you do (i.e. by getting project managers or peers to 
evaluate your project on them). No one project can excel on all of these 
simultaneously. 




There are three optional forms: 
Quality form 
Observation Form 



Artists/ Project Deliver's Feedback Form 

Sample questionnaires are given in Appendix Three. 



Quality Focus Group 

You could use a focus group instead of a questionnaire. Focus groups can 
be useful for providing feedback on the quality of your project. Guidelines for 
conducting focus groups are given in Appendix Two. The discussion can be 
informal if you are a small group. However think about how you might want to 
change this as you grow. For example, you might want to have separate 
sessions for artists and project managers. 
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1 




You need to revisit your People tasks during this stage: 
■ Encourage people to take part. 



■ Reassure them and encourage them to be honest. Explain the data will be 
confidential and only aggregate findings will be recorded. 



■ Is this the right activity for your organisation? Does it match your 
resources and abilities? Does it match your ideas for where the organisation 
is going? 

■ Are there any problems bubbling up? Are people, resources or groups 
of participants being stressed? Is the project sustainable? 

■ Are there long term tasks that are being neglected? Arts organisations 
are often over-stretched which means there is little time for research, 
developing new projects, advocating for your sector, finding out about new 
sources of funding or strategic planning. 

■ What have you learnt? Are there any barriers to applying your new 
knowledge? How do you ensure the lessons are applied? Could your lessons 
help other arts organisations? 

It is worth having an internal discussion about how the project should be 
improved. You can't improve all your work simultaneously. It is sensible to 
focus your effort. For example, you could identify two key improvements to 
make to your project if you re-run it. 

You might like to give individuals responsibility for following up specific areas 
of improvement. This could include doing research on the Internet or talking 
to other arts organisations . 



■ Ask for ideas of how they would improve the project. 




There are two Action tasks: 



Fine tune Your Project/Activity 

Think about these questions: 
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AT THE END OF THE PROJECT 

Evaluate outcomes 

Think About Your Outcomes 

Outcome measurement sounds intangible and difficult but you don't need a 
high level of detail, so it is relatively straightforward. For example, it is 
enough to ask whether participants increased their confidence as a result of 
the project, not by how much their confidence increased. 

The use of social outcomes is an attempt by funders to better match the 
objectives of arts organisations, to move away from a reliance on economic 
impact evaluation. It might sound mechanistic or instrumental, however, the 
voluntary and community arts do have strong effects on people. We hope 
that recording what already occurs can show the sector in a stronger light as 
well as opening some interesting discussions within the sector. 

All outcome evaluation should involve talking to project participants. 
However, you might also like to collect information from artists and other 
session organisers, partners and funders. The toolkit assumes that 
organisations working with children will ask for views on outcomes from 
artists and other partners rather than from the children themselves. 

You need to ensure that the effects can reas onably be said to come from 
your project rather than from other events in participants' lives. The approach 
that we have taken in the Outcome Questionnaire is to ask participants which 
effects come from your project. This is the simplest approach. You can also 
explore the role of your project by comparing participants ' questionnaire 
responses before and after the project, and by using a control group (similar 
group of participants who did not take part in your arts project). 

Projects often produce outcomes not anticipated in the funding proposal, 
positive and negative. Evaluation needs to be flexible enough to include 
these. 

If you want to ask participants for their feelings towards the wider community 
then you need to think carefully about which community you mean. People 
might trust others in their local neighbourhood but not trust people across the 
whole city. 





There are two optional forms: 
Outcome Questionnaire 

Sample questionnaires are given in Appendix Three. We have provided 
different forms for young people (11-16) and 1 7 and over. 

Follow up Questionnaire 

Arts organisations tell us that they do not have the resources to carry out 
follow up questionnaires. Therefore we have recommended that the Arts 
Council takes responsibility for the survey, which will be carried out through 
telephone interviewing. As with the Outcome Questionnaire a random 
sample of participants will be interviewed. This does not have to be the 
sample people as interviewed earlier, so long as the two groups are 
comparable. 

If your participants are younger than 1 1 then you can evaluate outcomes by 
asking for project workers to answer on their behalf: 

Artist's Evaluation Form 

Teacher's/Youth Leader's Evaluation Form 

Organisational Head's Evaluation Form 

Your should interview a sample from each of these three groups rather than 
just, say, artists, as this will give you greater confidence in your findings. 
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Revisit the People issues mentioned in the first section: 

■ Explain to those giving out the forms the importance of the information . 

■ Take the forms seriously. 

■ Encourage honesty. 

■ Collect a complete set of responses . 

■ Introduce the forms confidently. Don't be apologetic. 



There are two Action tasks: 
Interpret the Findings 

How good is good? It is arguable that all outcomes from arts projects are a 
bonus . However, it can also be interesting to compare your project with other 
projects from your or other organisations . What evaluators tend to find is that 
different types of projects consistently produce different kinds of outcomes 
and no project excels on all outcomes. 

Summary statistics can help you to compare your projects or your 
organisation with similar organisations , although they do not, of course, tell 
the whole story. You might like to look at unit costs, for example: 

■ The average cost per participant. The total cost divided by the number of 
participants. 

■ The average cost per session. The total cost divided by the number of 
sessions. 

■ The average cost per outcome. The total cost divided by the number of 
participants who experienced an outcome such as increased in confidence. 

■ The Arts Council subsidy per participant. The ACNI grant divided by the 
number of participants. 

■ The Arts Council subsidy per session. The ACNI grant divided by the 
number of sessions. 

■ The Arts Council subsidy per outcome. The ACNI grant divided by the 
num ber of participants who experienced an outcome such as an increase in 
confidence. 

Unit costs need to be interpreted alongside other figures, especially average 
contact hours and levels of outcomes . High unit costs can be good 
(indicating intensive support) or bad (indicating low participation or waste of 
resources). 



Look at the figures from the outcome evaluation and consider what they 
mean, for example: 



Were outcomes at a lower level 
than you expected? 


Why? 


Was this because the measurement didn't match 
your work, or because the take up/outputs were 
lower than expected, or because the quality was 
lower than needed? 


Were outcomes of the kind 
expected? 


Do you need to increase outreach? Do you need 
to fine tune publicity or delivery? 


Were there unexpected negative 
or positive outcomes? 


Do you need to expand your questionnaire to 
include unexpected positive outcomes? Do you 
need to change your project to reduce negative 
outcomes or to be more selective about 
participants ? 


Did some participants achieve 
higher outcomes than others? 


Do you need to target your project at participants 
who gain the most from it? 
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Act on the Findings 

Consider the following issues: 

■ Value for money. Are there ways you can increase the benefit without 
increasing the cost? Are there ways of decreasing the cost without 
decreasing the benefit? 

■ Was your logic model accurate and complete? Was the impact on the 
participants as expected? Were there any unintended positive or negative 
consequences? 

■ Was the project implemented as planned? Did the project reach 
targeted groups? If not, why not? Was the take up sufficient? If not, what 
could you do about it? Did partners do their bit? What were the critical 
success factors? What contextual factors helped or hindered the project? If 
the project was not implemented as planned, was this because of 
unanticipated problems? 

■ What have you learnt? What would you do differently next time? Most arts 
organisations have informal debriefing sessions . If you don't write the 
conclusions down then they might be lost. Evaluation gives a systematic 
form to do this. You can include recommendations about how funders can 
improve their programmes. 

■ What is the role of the project? What is unique about your project? 
Would the project suit other communities? Should it be rolled out or 
replicated? 



Evaluate impact 

There are three elements of impact: 

■ Organisational impact. This is the impact of projects or activities on your 
organisation's skills, reputation, innovation and its financial sustainability. 

Thinking . Social impact. This is the impact of projects or activities on the wider 

? ■ ? community, including but not limited to project participants and audience 

members. 

■ Economic impact. This includes the number of jobs created and income 
generated directly or indirectly from your organisation's work. 

If you have collected the information the toolkit recommends then measuring 
impact should be relatively straightforward. You should have most of the data 
you need. What remains it to : 

■ Add together your different projects. This needs to be done without 
double counting, which is relatively straightforward as your data are kept at 
the individual level. 

■ Take account of the role of other organisations . You should not take 
the credit for their work. In practice outcome measurement doesn't claim that 
arts organisations have sole responsibility for the outcomes. Rather it claims 
that arts organisations have contributed to positive outcomes. 

■ Consider external circumstances that affected the impact achieved. 

These could be circumstances that contributed to or counterbalanced your 
achievements. 

• Link to wider research about the impact of the arts on non- 
participants. The toolkit assumes that it is not the job of individual arts 
organisations to evaluate the overall impact of artistic activity on the wider 
community. 

These issues are considered in further detail in the review that was prepared 
alongside this toolkit. 
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There are two Systems tasks: 
Compiling Data for Social Impact 

All the outcome measures in the Outcomes Questionnaire can be turned into 
performance indicators of the form "Percentage of project participant who 
had x outcome". 



The Arts Council will compile three strategic performance indicators across 
organisations on: 

■ The percentage of participants in participative arts activities who increased 
in confidence (said that they increased in confidence as a result of the 
project). 

■ The percentage of participants in participative arts activities who increased 
in creativity (said that the project helped them to be more creative). 

■ The percentage of participants in participative arts activities who developed 
new skills (said yes to any of the skill options). 

Compiling Data for Economic Impact 

Economic impact is likely to be carried out by an external evaluator. 
However, there is a lot you can do to compile the data they need for their 
analysis, for example: 

• Keeping clear accounts of income and expenditure. 

• Keeping figures on the people employed on your projects. This 
information is on the Activity Form for Participation. 

• Noting whether your suppliers are local or outside your area. 




There is one People task: 
Discuss Organisational Impact 

Did the project develop or deplete your organisation's skills, partnerships and 
resources? 



There is one Action task: 




You might like to compare your results with other arts organisations to see if 
you can learn from each other. Evaluators tend to find that all projects have 
strengths and weaknesses. 
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APPENDIX ONE: REFERENCE 

What are the types of evaluation? 

Summative evaluation typically takes place at the end of your project and 
makes a judgement about the overall success of your work. If you think of 
goals as the overall effects you wish to achieve from your project, then 
summative evaluation examines your progress towards your goals. 
Summative evaluation typically looks at outcomes and impact. 

Formative evaluation takes place during your project and provides feedback 
on whether you are making progress. If you think of objectives as the means 
to achieve your goals, then formative evaluation examines your progress 
towards your objectives. Formative evaluation typically looks at process 
measures, outputs and intermediate indicators. 

What are the advantages and disadvantages of doing 
evaluation in-house versus using external evaluators 

The advantages of doing evaluation in-house are: 

■ Knowledge of internal culture and process. You might have a better 
understanding of the information that you need as well as the practicalities 
of applying evaluation in your organisation. 

■ Organisational learning. Learning about evaluation techniques and 
insights gained from the evaluation stays in the organisation. 

■ Commitment. You might be in a stronger position to implement 
recommendations that come out of your evaluations. 

The advantages of using external evaluators are: 

■ Specialist skills. An external evaluator might have long experience in 
tasks such as writing and analysing questionnaires. 

■ Independence. Reports produced by external evaluators might have 
greater credibility with funders. 

■ Distance. An outsider might bring a fresh perspective to your work. 

■ Knowledge of other areas. An external evaluator might know about good 
practice in other areas, whether in evaluation or project delivery. 

In practice, your access to external evaluators is likely to be determined by 
your resources. If you wish to use external evaluators then you will need to 
build their fees into your budget from the beginning. 

What are the strengths and weaknesses of different 
evaluation methods? 

There are seven broad methods available. Which evaluation method is 
appropriate will depend on: 

■ The type of questions to be answered. 

■ The time available. 

■ The resources available. 
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■ The technical skills of the evaluation team. 

■ The intended audience for the evaluation and what methods they consider 
credible. 

Evaluation methods can be broadly classified into: 

■ Quantitative methods. Data that can be represented numerically. 

■ Qualitative methods. Narrative descriptions or images. 

These are better thought of as two ends of a continuum rathe r than two 
discrete categories. Open questions, which are by nature qualitative, can be 
coded to produce quantitative data. Closed questions, which are by nature 
quantitative, can include space for comments. The two are complementary. 
Quantitative data helps with generalisation. Qualitative data helps with 
interpretation. 

Arts organisations often feel that evaluation is mechanistic because it relies 
on numbers. There is an assumption that numbers are restrictive, simplistic 
and rigid, but this does not have to be the case. In fact, numbers can be 
deeply creative, radical, even anarchic. It is a matter of fact that the human 
brain works by simplifying reality: it sees what it expects to see or 
concentrates on the most vivid or recent occurrences. Numbers can help us 
see past our own expectations or experiences. When properly used, numbers 
tell the story as it is. Numerical data helps organisations to detect trends or 
occurrences that are not obvious on their own. Thus, instead of just drawing 
attention to what the most vociferous participants think, numbers can tell an 
organisation what the silent majority think. 

There are usually trade-offs between breadth and depth. All methods have 
strengths and weaknesses. Ideally you should use two or more methods 
which have complementary features. This is described as "triangulating" on 
the research subject because it gives you different perspectives, rather like 
taking spatial measurements from two different points. 

The task lists given in this section do not include the generic tasks, such as 
notifying respondents, which are relevant to all evaluation methods. 

Observation 

Observation is a holistic method that can give the evaluator a vivid picture of 
the operation of a project, especially with regard to group behaviour. The 
complexity of observation is a weakness as well as a strength. Some form of 
evaluation questionnaire is helpful to steer observation. Otherwise there is a 
distinct risk that different observers would come to different conclusions (low 
reliability). 



Strengths and weaknesses of using observation 



Strengths 


Weaknesses 


Dense information source 


Small sample of behaviour 


Immediate 


Lack of understanding of preceding events : can be 
difficult to interpret or categorise seen behaviours 




Reliance on observer's skills 




The visit alters behaviour (the Hawthorne Effect) 



The tasks in organising an observation session are: 
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■ Consider the aims of the observation sessions and the information that 
you would like to obtain. It might be helpful to think about what you would 
expect to see (good behaviour) and what you would not expect to see (bad 
behaviour). 

■ Consider the number of sessions that you would like to have observed and 
the length of observation. This is likely to be determined by your 
resources. 

■ Think about a general sampling framework: whether you would like to 
include different types of activities, different staff members, different types 
of participants or different times of year. 

■ Notify the session organisers about the planned observation and think 
through the logistics, for example, whether the observer should take an 
active part in the session. 

■ Produce an observation schedule. Drafts are given in Appendix Three. 

■ Observe the session and take notes during or immediately after the 
session. 

■ Discuss any queries or other relevant issues with the session organiser. 

■ Produce a short summary of the findings from observation sessions over a 
set period of time e.g. six months or a year. 

Written or emailed surveys 

Written surveys are useful for compiling simple factual information across a 
relatively homogeneous population (homogenous in terms of background or 
outcomes). They are particularly successful where the subject is well 
understood so that the questions can be coded. They are not applicable for 
complex or conceptual issues. Nor are they suited to situations where 
respondents vary widely and would therefore need to answer different parts of 
the questionnaire. The response rate can be maximized by personalising the 
letters and questionnaires (including the name of the person addressed and a 
reference to the event or project covered), including stamped addressed 
envelopes, and issuing at least one reminder letter or email. Written 
questionnaires should always be tested using a small pilot survey, which can 
be administered face-to-face. Postal surveys demand more skill in 
questionnaire writing than telephone or face-to-face surveys because they 
lack the opportunity to check responses with respondents. 



Strengths and weaknesses of using written surveys 



Strengths 


Weaknesses 


Cheap 


Low response rate | 


Large numbers do not greatly increase the cost 


Biased response 


Provide uniform information 


No assurance the questions were understood 


Data entry is simple 


No assurance the addressee was the one who replied 


Can be anonymous 


Responses can be difficult to interpret 


Self-administered 


People often express themselves better orally than in 
writing 




Limited personal contact to motivate a response 




Assume a level of literacy 


The tasks in organising a postal survey are: 
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■ Consider the aims of the survey and the information that you would like it 
to address. 

■ Check your contact details for planned respondents. Correct any 
information that is incomplete or out of date. 

■ Draft the survey questionnaire. 

■ Pilot the survey questionnaire by getting three or four potential 
respondents to fill it in and then tell you if the questions were clear, 
applicable and in the right order. 

■ Amend the survey questionnaire with insights from the pilot. 

■ Write a covering letter explaining the aims of the survey and the purpose 
to which information will be put. Include reassurances about confidentiality. 

■ Copy and post/distribute the survey questionnaires. Consider including 
stamped addressed reply envelops. 

■ Send out a reminder, as appropriate. 

■ File and number questionnaires as they come in. 

■ Produce a database structure, enter and analyse the questionnaires. 

Face-to-face surveys 

Face-to-face surveys provide detailed, personal interaction with interviewees. 
This is a strength because of the potential for empathy and probing to explore 
answers and a weakness because of the scope for personality conflicts and 
manipulation. Face-to-face surveys are expensive and best applied when the 
number of people to interview is relatively small or concentrated in one area. 



Strengths and weaknesses of using face-to-face surveys 



Strengths 


Weaknesses 


Personalized 


Expensive I 


In depth, free response 


Time consuming 


Empathy can motivate a longer and more complete 
discussion 


May intimidate some groups 


Flexible and adaptable 


Open to manipulation by the interviewer 


Gives visual cues 


Vulnerable to personality conflicts 


Can combine open questions with pre-coded 


Requires skilled interviewers 


responses 






Might be difficult to summarise findings 




Might involve the interviewer travelling to unsafe areas 



The tasks in organising a face-to-face survey are: 

■ Consider the aims of the survey and the information that you would like it 
to address. 

■ Consider whether you need a full response or a sample and if the latter, 
what size sample is appropriate. All samples must be random (e.g. every 
10 th participant) not personally selected. 

■ Draft the survey questionnaire. 
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■ Pilot the survey questionnaire by getting three or four potential 
respondents to fill it in and then tell you if the questions were clear, 
applicable and in the right order. 

■ Amend the survey questionnaire with insights from the pilot. 

■ Fix appointments with the planned respondents and carry out the 
interviews. You should take full notes during the interview to complete the 
questionnaire but also record any vivid comments and new issues raised. 

■ Check the questionnaire at the end of each interview to ensure that all 
questions are answered and the writing/coding is intelligible. 

■ Produce a database structure, enter and analyse the questionnaires. 
Telephone surveys 

Telephone surveys combine some of the advantages of postal surveys 
(relatively low cost) with some of the advantages of face to face interviews 
(personal interaction). 



Strengths and weaknesses of using telephone surveys 



Strengths 


Weaknesses 


High response rate 


Can be difficult to contact people 


Empathy can motivate a longer discussion 


Problem with time differences 


Scope for checking meaning 


Inappropriate for people without telephones 


Scope for following leads 


Inappropriate for children and non-English speaking 
people 


Cheaper than face to face 


Respondents might lack privacy [ 


Interviewee is relaxed in their own environment and 
tends to be more candid 




Lead-in time to start the survey is short 




Contact with the interviewee is used productivity rather 
than in chasing up written forms 




Can combine open questions with pre-coded 
responses 





The tasks in organising a telephone survey are: 

■ Consider the aims of the survey and the information that you would like it 
to address. 

■ Draft the survey questionnaire including introduction and confidentiality 
undertaking. 

■ Pilot the survey questionnaire by getting three or four potential 
respondents to fill it in and then tell you if the questions were clear, 
applicable and in the right order. 

■ Amend the survey questionnaire with insights from the pilot. 

■ Telephone the respondents. If you are telephoning using a random survey, 
make at three attempts to telephone each numbered respondent. Track 
planned interviewees if their contact details appear to be out of date. 

■ Check the questionnaire at the end of each interview to ensure that all 
questions are answered and the writing/coding is intelligible. 

■ Produce a database structure, enter and analyse the questionnaires. 
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Focus groups 

Focus groups collect data through group interaction on a topic determined by 
the researcher. Focus groups provide an excellent method for seeing how 
your clients think and what issues are important to them. They help to 
generate questions but not answers. Results need to be tested using a larger 
survey to produce generalisations. The group setting means that individuals 
have to wait for a time to talk which can reduce rather than increase the 
number of ideas generated. The value of focus groups is strongly affected by 
the skills of the facilitator. 



Strengths and weaknesses of using focus groups 



Strengths 


Weaknesses 


Group interaction 


Small sample 


Group consensus 


Not representative/random | 


In depth discussion 


Responses are not independent: 

Group format fosters conformity: differences might be 

suppressed or polarized 

Vulnerable to manipulation by a skilful member. 

Affected by group cohesion (compliance, identification, 

internalization) 


More efficient than one-to-one interviews 


Questions are not asked the same way each 
time/research method is not replicated 


Democratic process in having the 
researcher/moderator outnumbered by the participants 


Difficult to quantify 


Relatively immediate 


Not suited to potentially sensitive issues 



The tasks in organising a focus group are: 

■ Clarify the purpose and questions to ask. 

■ Think about the number of focus groups to run. 

■ Decide on and invite participants. A general guideline is to have eight to 
ten participants per focus group. 

■ Book a room. It should be accessible, of the right size and private. The 
group should not be disturbed during the session. 

■ Organise and brief the facilitator. 

■ Organise and brief the note-taking. 

■ Organise the room. Ensure that seating promotes group discussion e.g. in 
a circle. If some people do not turn up remove empty seats. 

■ Ensure the meeting starts and finishes on time. You might like to set some 
ground rules for the group, for example, participants should talk to each 
other not the moderator, there should be only one conversation at a time 
and everyone should have a say. Remember that the role of the meeting is 
to understand how the participants think and what matters to them, not to 
reach a consensus. 

■ Analyse the results. Describe the nature of the discussion and identify 
patterns or themes. A summary should be written from this account, not 
from memory. It is better to write up notes from the meeting while the 
impressions are still fresh rather than some time later. 
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Case studies 

Case studies provide a detailed and real world method that is complementary 
to wider, shallower methods such as postal or telephone questionnaires. They 
can show how processes work over time and therefore provide insight into 
causation. There are two types of case studies used in evaluation. 

■ Detailed case studies can be used to provide a holistic picture of projects, 
especially to check the mechanisms whereby effects were achieved. 

■ Brief case studies can be used to illustrate evaluation reports by giving 
vivid examples. 

Organisations or individuals are usually chosen for case study because they 
are interesting or exceptional in some way. Subjects are not usually typical 
(representative) and therefore case studies should not be used to make 
generalisations about the wider group. 



Strengths and weaknesses of using case studies 



Strengths 


Weaknesses 


Multi-faceted: can show different perspectives 


Time consuming 


Vivid 


Expensive 




Anecdotal 



The tasks in producing case studies are: 

■ Identify criteria for case studies. For example, the issues or subject areas 
that you wish to illustrate. 

■ Draft a structure for the case studies. This should include the headings 
under which text will be presented and the questions that need to be 
asked to generate the information needed. 

■ Produce a long list of possible case study individuals/organisations. 

■ Produce a grid of each possible case study against the selection criteria. 

■ Select a shorter list of case studies that covers the selection criteria. 

■ Contact the short list individuals/organisations and check that they are 
interested in being interviewed. Fix specific dates for the interviews. Offer 
any reassurances that they need, such as that they can see and comment 
on a draft of the text. For longer case studies, any written or visual material 
needed could be requested at this stage. 

■ Interview the case study individuals or organisations, being careful to take 
notes of the conversations. Check over your notes immediately after the 
interview to ensure that all key points are included. 

■ Write up your notes into the case study structure. 

■ Send your draft case studies to the subjects for comment. 

■ Include any comments that do not distort the meaning. Any factual errors 
should be corrected, but the neutral tone of the text should be maintained. 
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Art works, video and film 

Artworks, video and film can provide evidence of the achievements of a 
project. Visual materials are vivid, but not always easy to interpret. They can 
be enhanced by some form of structure or commentary that explains how the 
group worked, the role of the artists versus the participants and the level of 
skill that participants had before joining the project. They can also be more 
effective if kept short. 



Strengths and weaknesses of using video and film 



Strengths 


Weaknesses 


Vivid 


Expensive 


Consistent with artistic values 


Impression can be affected by the quality of filming 
rather than the quality of the project 


Motivating for participants 


Time consuming | 




Dependent on the skills of the viewer 



Checklist of tasks for producing video or film evidence: 

■ Obtain the necessary permissions from individuals you are planning to 
film. 

■ Consider including an introduction describing the content and summarising 
the main conclusions from the video. This will motivate the viewer to look 
at the video and provide them with key information if they do not watch the 
whole thing. 

■ Consider including interviews with key stakeholders rather than just a film 
of the final presentation. 

Records 

Records are existing sources that have been compiled for internal 
management uses. Examples include application data and client databases. 
Arts organisations are not likely to use records as a source of information for 
their evaluation: this heading is included purely for the sake of completeness. 

Records can be useful for giving background information. At their best they 
give an historical series of data that is difficult for an evaluator to compile after 
the event. At their worst they are inflexible, incomplete, inconsistent and 
subject to hidden rules about how the data are recorded. 



Strengths and weaknesses of using records 



Strengths 


Weaknesses 


Cheap 


Often incomplete, inaccurate and out of date 


Time series 


Not usually compiled for this purpose i 


Non reactive 


May have confidentiality restrictions 


Do not interrupt the project 


Changing rules might make comparison invalid 


Set down at the time of the project not retrospectively 


Can be misleading unless the rules are explained I 




Factual only: no values or attitudes 



The tasks in using records are: 

■ Check that you understand the purpose for which the records were 
compiled and any limitations that arise from this intended purpose. 
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■ Look through each field of data in order to identify any gaps or anomalies. 

■ Produce tabulations for each field of data. 

■ Allow time to read and reflect on the tables. 

■ Produce cross-tabulations for any areas of particular interest in the data. 
How do I write questionnaires? 

Questionnaires are helpful for all evaluation methods. Following up or clarify 
comments made by respondents can be useful, but a totally free form 
interview can end up concentrating on recent or memorable issues and 
thereby miss out other important subjects, (for example those that are taken 
for granted). 

Questionnaires ensure that: 

■ You are clear about which issues you wish to investigate. 

■ Questions are not missed out by accident. 

■ Each individual or group is interviewed in the same way. 
There are four stages in writing a questionnaire: 

■ Think about the issues that you need to investigate. 

■ Translate issues into questionnaires. 

■ Fine tune each question. 

■ Check the layout and presentation of the questionnaire. 

Defining evaluation issues 

Evaluation issues should be: 

■ Important. It is better to address a few key issues well rather than to 
dissipate energy attempting to pursue all the issues relevant to your 
project. 

■ Precise. It might take some time to see the exact issue that you want to 
investigate. For example, a question "did we reach our target 
participants?" might be honed down to a question "did boys join and stay 
with our project?" 

■ Relevant to you. Some of the broader or more theoretical questions might 
be better answered by Arts Council research or by a joint evaluation 
project shared by several arts organisations. 

■ Actionable. Think about what you will do with the answers. Don't ask for 
information that you can't use. 

■ Rounded. Include "why" and "how" questions as well as "what" and "how 
many". 

Translating issues into questions 

The issues that you want to investigate need to be translated into questions to 
be posed to interviewees. All evaluation methodologies benefit from a clear 
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understanding of the questions to be answered. Guidelines for translating 
issues into questions include: 

■ Write questions from the point of view of the respondent. Ensure the 
wording and focus is related to their objectives. Don't ask respondents to 
express an opinion on which they have no evidence. It is generally not 
good practice to ask respondents to tell you their opinion of the opinions or 
experiences of other people. The exception to this rule is for teachers in 
educational projects. 

■ Use mainly closed rather than open questions. A list of specific 
questions about the project is better than one question about "What did 
you like about the project". The latter is ambiguous and dependent on the 
perception, memory, character and mood of the respondent. Closed 
questions provide more usable information and are faster for the 
respondent to complete. Include one or two open questions at the end, 
such as what they liked or what they think should be changed, so that you 
can find out about any issues that you might have missed in the specific 
questions. 

■ Keep the questionnaire short. One to two pages should be sufficient. 
Writing questions 

A good question is one that is insightful rather than trivial, and one that can be 
communicated precisely to the people from whom an answer is expected. 
Check each question against this checklist: 

■ Ensure questions are clear. Do not use double negatives. Do not use 
questions with complicated qualifications. Do not use jargon or 
abbreviations. Use plain English. 

■ Ensure questions are precise. Do not ask about two things at the same 
time. Give time periods if necessary. 

■ Ensure questions are relevant to the respondent. Do not use questions 
based on suppositions. Do not use questions that assume respondent 
have knowledge that they might not have. Do not ask respondents to 
speculate about other people's attitudes or behaviour (except for youth 
arts). 

■ Ensure questions are simple. Do not ask respondents to perform 
calculations - ask for the data and calculate the sums yourself. Do not ask 
questions requiring generalisation or memory. Do not ask questions that 
require respondents to summarise events that happened over a long 
period of time. Do not ask questions that require respondents to remember 
events that happened a long time ago. 

■ Ensure questions are neutral. Do not use loaded questions. Do not infer 
that a particular answer is expected. Always keep response codes 
balanced (an equal number of positive and negative codes). 
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Balanced response codes 



29. Would you like to be involved in 
more of the organisation's projects? 


Definitely yes 
Probably yes 


Definitely no 
Probably no 


Imbalanced response codes 


29. Would you like to be involved in 
more of the organisation's projects? 


Definitely yes 
Probably yes 


No 

Don't know 



Proof read the questionnaire. Get other people to read it through to check that 
it isn't ambiguous. You should also pilot the questionnaire: ask three or four 
participants to fill it in and then tell you whether the questions seemed clear, 
appropriate and complete. Piloting is necessary: 

■ To check that the questions produce the information needed. 

■ To check that interviewees understand the questions. 

■ To check that the questionnaire can be applied in the context intended. 
E.g. how long it takes. 

Checking the layout and presentation of the questionnaire 

It is important that you obtain complete questionnaires where each question is 
answered and respondents are motivated to reply honestly and with thought. 

■ Introduce the questionnaire. Motivate respondents to complete the 
questionnaire. You should explain why the information is important: 
generally that it will help your organisation to improve. You should also 
reassure the respondent that individual responses are confidential. 

■ Number the questions. This makes it easier for the respondent to see 
when they have missed out a question. 

■ Number coded questions. The data entered can then type in responses 
straight from the questionnaire. 

■ Ensure the respondent has enough information to answer each 
question. Tell the respondent whether they should tick one answer only, 
tick all answers that apply to them or write an answer in the box. 

■ Ensure the order is appropriate. A common approach is to start 
questionnaires with factual, uncontentious subjects that get the 
interviewee talking/thinking, then to move on to more intangible questions 
about feelings or attitudes and finish with personal/demographic questions 
which are often perceived as boring or intrusive. 

■ Ensure that you have routed respondents past irrelevant questions. If 

an answer to one question renders the rest of that section irrelevant then 
direct the respondent to the next section. 

■ Ensure that there is enough room for responses. This is especially the 
case for open questions. 

■ Ensure the structure is logical. Group similar questions together. Ask 
general questions after specific, so that you have already led the 
respondent to think about the issue before putting forward overall views. 
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■ Ensure the questionnaire is easy to read. Avoid changing the response 
type. If you are going to use "agree" "disagree" as well as "yes" "no" 
responses codes then try not to skip back and forth between the two. 

■ Check page breaks. Do not break questions in the middle. 

■ Thank the respondent. 

■ Remind the respondent about how they can return the questionnaire. 

Even if you are planning for respondents to fill the questionnaire during a 
session, you should still include your address on it in case they take the 
form home. 

The layout we use is designed to simplify data entry: the person entering the 
data onto the computer only has to read the right hand column instead of 
having their eyes darting all over the page. 

How do I draw a sample? 

The toolkit assumes that all samples should be random. Random sampling 
reduces the bias in your survey and therefore reduces the need to interview 
large numbers of participants. 

The procedures for drawing a random sample are: 

■ Decide on the total number of responses that you would like. This should 
take account of your resources, but also of the types of questions you 
would like to ask. If you would like to compare different groups of 
participants (e.g. different ages, sexes, projects) then you ideally need to 
have at least 30 respondents in each sub-category. 

■ Translate the number of responses into a number of questionnaires or 
interviewees. Postal surveys can have very low response rates, so you 
might need to send out 5 or 1 times the questionnaires you need. 
Telephone surveys have high response rates and you only need 2 times 
the target response number on your contact list. 

■ Select your contact list randomly. For instance, if you have twice as many 
participants as you need to contact then choose every second participant 
or every other project. 

■ Make three or four attempts to contact each person on your contact list. 

■ When you have finished the survey, compare the characteristics of your 
sample with the total population of participants. For example, does your 
sample have more older participants in it? If younger participants are 
under-represented in your survey, then this will affect the kind of claims 
that you can make about your results. 

How do I analyse data? 

Analysing data is a bit like doing a jigsaw puzzle in which the pattern gradually 
emerges. There are five stages to analysis: 

■ Enter the data. It is relatively easy to construct a spreadsheet with 
columns for questions and rows for respondents. If you have pre-coded 
your responses then all you will need to do is to key the response code 
(response a,b or c etc) into the spreadsheet or database. Open questions 
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might need to be classified before they can be recorded, or to be 
summarised as a text field. 

■ Clean the data. Read through the questionnaires and check for any 
answers that are inconsistent or unlikely. If there are anomalies then check 
the questionnaires to see if this is a coding error. 

■ Analyse the data. You might have to analyse the data in stages. Start by 
producing tabulations and averages. These might suggest further 
questions, such as whether some types of sessions were more likely to 
produce some types of outcomes. You can follow these questions by 
doing cross tabulations. Be careful that you have enough data in each 
subgroup. A rule of thumb is that a minimum of 20 cases is needed in 
each subgroup. 

■ Reanalyse the data. Note any new ideas, patterns or themes in the first 
analysis, for example, whether different groups hold different views, 
whether different outcomes peak at different times in the project or 
whether different types of participants have different outcomes. You might 
want to follow these up with further analysis. 

■ Present the data. You should include tables of the survey data, whether 
in the main report (for key issues) or in an appendix (for background 
questions). Provide enough information for the reader to understand the 
data. For example, you should state the number of respondents to each 
questions. Avoid using percentages where the total number of 
respondents is less than 20. You can use quotations of comments made 
by interviewees to communicate key points in the data. 

Don't throw away the data after the report has been written. You might like to 
re-analyse the data in the future as new issues emerge. 

How do I write an evaluation report? 

Evaluation reports need to be clear, coherent, rigorous and concise. You need 
to think about the style , content and process. 

Style 

With a bit of effort you should be able to write an evaluation report that looks 
professional. Here are a few guidelines on style: 

■ Be precise. Use the active not the passive voice: say "the project co- 
ordinator ran a follow up session" not "a follow up session was run". In the 
second the reader is left wondering if your organisation or someone else 
ran the follow up session. Avoid vague terms like "some". 

■ Concentrate on fact. Describe what happened clearly. Keep speculation 
about what might have happened or will happen in the future to a separate 
part of the report. If the two are mixed up the reader might lose sight of 
what you actually did. 

■ Be balanced. Totally positive reports are not credible. Don't forget to 
celebrate the strengths of your project in full - don't take them for granted. 
However, also be honest about the weaknesses of your project. Explaining 
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problems that you have had can be one of the best ways of arguing for 
increased funding. 

■ Don't shy away from using numbers. A little quantitative information can 
be vivid and can save you a lot of writing. Evaluation reports work best 
where they combine numbers and discussion. 

■ Respect confidentiality. Make sure that quotations or comments cannot 
be traced back to individual respondents. If there is a possibility that 
respondents could be identified, then you should get their permission to 
include their comments. 

■ Depersonalise criticism. Talk in terms of the tasks and outcomes that 
were not achieved, not the people who failed. There are many reasons for 
outcomes to be missed that are not connected to the individuals 
concerned (for example, poor project logic, resource constraints and 
changes in the environment). 

■ Integrate data sources. Your main report should bring together findings 
from different evaluation methods to address your research questions. Use 
both figures and words to describe your findings. 

■ Reconcile inconsistencies. Sometimes different evaluation methods 
yield different results. Where this happens, you should give an explanation 
for the discrepancy, whether due to weaknesses in one set of data or 
differences in the circumstances of the two sets. 

■ Take a dynamic view. Evaluation tends to take a snap shot at one period 
in time. In your interpretation, you should place this snap shot in context. 
Remember that many of the subjects you are measuring, especially 
outcomes, change over time. Some outcomes (such as confidence) tend 
to go down during a project, as participants increase in self awareness and 
raise their aspirations, and then peak at the end of the project, especially 
after a final performance. Some outcomes (such as social outcomes) take 
time to build up and might peak some time after the project has finished. 

■ Use your logic model. You might like to consider where in the chain of 
reasoning any weak links lay. For example, was the project not 
implemented as planned because of resource constraints or were the 
outcomes not achieved because the logic was faulty? 

Content 

There are really only four broad questions that you need to answer in your 
evaluation report: 

What did you do? 

You need to describe what you did and what other partners did. 
This section could include: 

■ What were the objectives for the project? 

■ What were the needs that the project was intended to address? 

■ Who was involved in the project? 

■ Who organised the project? 
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■ Who initiated the project? 

■ Who funded it? 

■ What was the project? 

■ When did the project take place? 

■ What activity was carried out? 

■ What were the resources (financial and non financial) involved in 
delivering the project? 

What did you achieve? 

This should be mainly about the benefits to participants, but also include 
benefits to your organisation such as capacity building. 

This section could include: 

■ How many participants took part? 

■ What was the profile of participants (age, sex, location, ethnicity, indicators 
of social exclusion)? 

■ What were the outcomes for participants? 

■ Did some participants benefit more than others? 

■ How were these outcomes achieved? 

■ Were any problems experienced in delivering the project? 

What is the evidence for those achievements? 

This should consider the evidence that benefits are due to your project rather 
than to other events that happened to the participants. You don't need to have 
a complicated experiment with control groups to prove cause: it helps if you 
can explain the mechanism for the effect: what it was about your project that 
produced the stated benefits. 

This section could include: 

■ What evaluation was carried out? 

■ How were outcomes measured? 

■ How were data analysed? 

■ How were the data interpreted? 

■ What are the conclusions from the evaluation? 

■ What steps did the evaluation take to avoid bias? 

■ What are the limitations o n the evaluation methods? 

What are the implications for future work? 
This could include: 

■ Were the objectives appropriate? Did they change during the project? 

■ Was take up as high as it should have been? If not why not? 
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■ What key decisions were made in the project? What criteria were applied? 
In retrospect, should different criteria have been applied or the same 
criteria applied differently? 

■ What was good about the project? 

■ What lessons should be applied in future projects? 

■ Where there further questions that the evaluation could not answer? Is 
further work needed? 

You should include a summary that picks up the key points in this list. You 
can include copies of survey questionnaires, data tables and lists of 
organisations consulted (if relevant) in appendices. 

Process 

It is worth producing a draft and getting colleagues to comment on it. 

How do I disseminate an evaluation report? 

General guidelines are: 

■ Evaluation reports should be freely available. For example, if you have a 
website, you might like to make full reports or summaries available on your 
website. 

■ Customise reports to different audiences. 

■ Deliver reports in time to be useful. 

■ Use different media, for example conference presentations and workshops 
as well as written reports. 

■ Use partners' communication media, for example include a summary of 
your evaluation in their newsletters. 

■ Remember to communicate your findings to respondents who participated 
in your evaluation. This is courteous and helps to ensure their cooperation 
in future work. 

What Is a good evaluation ? 

A good evaluation should be: 

■ Meaningful. 

■ Accurate. 

■ Fair. 

■ Useful. 

■ Practical. 

The last four of the five headings map onto the criteria advocated by the 
American Program Evaluation Standards (Joint Committee on Standards for 
Educational Evaluation, 1994): accuracy, propriety, utility, and feasibility. 
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A good evaluation is meaningful 

Evaluation should focus on the most important and topical issues facing the 
organisation, not just the ones that are easiest to measure. A good evaluation 
helps the organisation to think about what they are doing, the assumptions 
they are making, and the different perspectives of different stakeholders. A 
good evaluation is clear in its thinking. 

You can help to make the evaluation meaningful by: 

■ Thinking about why you are doing the evaluation. It is not a good use 
of money to employ evaluators for questions if the answers will not change 
the organisation's way of working. 

■ Being focused in your objectives. Evaluations tend to work best if they 
focus on a small number of connected issues rather than a ragbag of 
unconnected issues. 

A good evaluation allocates time for thinking about which questions should be 
explored before starting the research. 

A good evaluation is accurate 

Accuracy standards require an evaluator to base their conclusions on sound 
information. To meet these standards, you should: 

■ Describe your methods in your evaluation report. The methods and 
findings of the evaluation should be presented in enough detail in the 
report so that the adequacy of information can be judged. It is not sufficient 
to present the conclusions without explaining how they were derived. 

■ Choose and apply methods to ensure the validity and reliability of 
data. You should not place reliance on biased, partial or spurious data. 
Survey data should not be derived from a small or biased sample. 
Questions should not be leading or loaded. 

■ Analyse the findings systematically. Each piece of information should 
be examined and interpreted. 

■ Explain and justify conclusions. You should describe your conclusions 
separately from your findings in such a way that the link between the two 
is evident. 

■ Report findings and conclusions impartially. The language or 
presentation of the report should not presuppose a particular conclusion. 

A good evaluation has a logical track from the choice of methods, through the 
results to the conclusions. 

A good evaluation is fair 

Propriety standards require evaluators to be lawful and ethical, respecting the 
rights of people involved in the evaluation. 

You should : 

■ Respect the dignity of human subjects. Evaluators should not be 
adversarial or blinkered. 
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■ Report the conclusions in a fair and balanced manner. Most activities 
have strengths and weaknesses. An evaluation report should comment on 
both. 

■ Deal with any conflict of interests openly and honestly. Evaluators 
should not hide or ignore any conflicts of interests they have in evaluating 
the organisation. 

Evaluations are expected to draw their conclusions independently, without 
any external pressure. 

A good evaluation is useful 

Utility standards require evaluators to be informative, timely and influential. 
A good evaluation: 

■ Considers the intended use for the report from the beginning. For 

instance, the evaluator should ask about whether the report is aimed to 
provide general insight about the area of activity, or specific guidance 
about improving practice. 

■ Keeps the client informed about decisions and emerging issues in 
the evaluation. A "no surprises" approach works best. 

■ Presents relevant information clearly and in a timely fashion. 

Recommendations should be practical and able to be implemented. 

It is good practice to provide different presentations of the evaluation for 
different audiences. 

A good evaluation is practical 

Feasibility standards require evaluations to be realistic, prudent, diplomatic, 
and economical. They specify that evaluators should cause the minimum 
disruption to the program being evaluated, should be sensitive in their 
involvement of stakeholders, and should be mindful of the cost of the methods 
they are proposing. 
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APPENDIX TWO: OBLIGATORY QUESTIONNAIRES 

You should customise these questionnaires by putting the name of your 
project and your organisation instead of "The project" and "The organisation". 
You might also like to change the font size to make the questionnaires easier 
to read. If you amend the questionnaires please check the page breaks before 
you print the questionnaires: you should never break a question in the middle. 

Obligatory questionnaires are needed to collect performance indicators for the 
Arts Council. 

Activity Forms must cover all of the work that your organisations carries out 
with any Arts Council funding. 

NOTES FOR THE ORGANISATIONAL FORM 

The Organisational Form gives background information that enables the Arts 
Council to understand the results from different types of arts organisations, for 
example to different art forms, sizes or ages of organisation. 

You only need to fill the form in once. Thereafter, you only need to update 
fields that have changed. 
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ORGANISATIONAL FORM 



1 . Name of organisation 




? Datp of p^tahlicihmpnt 

£— . L/ULC Ul L CI kJIIOI II 1 1^1 1 L 




3. Art form 


a. Music 

b. Drama 

c. Dance 

d. Literature 

e. Visual arts and 
crafts 

f. Film and video 


g. Architecture and 
murals 

h. Traditional arts 

i. Combined arts 
j. Other, please 
specify: 


4. Type of organisation 


a. Mainstream 

b. Community and voluntary 

c. Amateur 


5. Legal structure 


a. Unincorporated club or association 

b. Company limited by guarantee 

c. Company limited by shares 

d. Recognised charity 

e. Trust 

f. Local Authority 

g. Other, please specify: 


6. District Council of home base 


a. Antrim 

b. Ards 

c. Armgagh 

d. Ballymena 

e. Ballymoney 

f. Banbridge 

g. Belfast 

h. Carrickfergus 

i. Castlereagh 
j. Coleraine 

k. Cookstown 
1. Craigavon 
m. Derry 
n. Down 


o. Dungannon 
p. Fermanagh 
q. Lame 
r. Limavady 
s. Lisburn 
t. Magherafelt 
u. Moyne 

v. Newry and Mourne 

w. Newtown abbey 

x. North Down 

y. Omagh 

z. Strabane 

Other, please specify: 


7. Parliamentary Constituency of home 
base 


a. East Antrim 

b. North Antrim 

c. South Antrim 

d. East Belfast 

e. North Belfast 

f. South Belfast 

g. West Belfast 

h. North Down 

i. South Down 

j. Fermanagh/South 
Tyrone 


k. Foyle 

1. Lagan Valley 

m. Londonderry East 

n. Newry & Armagh 

o. Strangford 

p. Mid Ulster 

q. Upper Bann 

r. West Tyrone 

s. Other, please 

specify: 


8. Type of organisation 


a. Gallery 

b. Producing venue 

c. Receiving and producing venue 

d. Touring company 

e. Festival 

f. Community arts organisation 

g. Disability arts organisation 

h. Publisher 

i. Umbrella/arts resource organisation 
j. Other, please specify: 



Annabel Jackson Associates 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 

51 



9. Total number of staff (full time 
equivalents) 




1 0. Number of full time artistic staff 




1 0. Number of part time artistic staff 




1 1 . Number of full time executive staff 




1 2. Number of part time executive staff 




1 3. Number of other full time staff 




14. Numuer ot otner part time start 




1 5. Number of volunteers 




1 6. Number of staff who are disabled 




1 7. Number of staff from ethnic 
minority groups 




1 8. Is the organisation or its 
programme disability led? 


a. Programme disability led 

b. Organisation disability led 



Annabel Jackson Associates 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 



52 



NOTES FOR THE SUPPORT SERVICES FORM 

The Support Services Form has been introduced to ensure that the Arts 
Council has some account of the work carried out by umbrella and other 
organisations, which will not show up in the Activity Forms. 
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SUPPORT SERVICES TO ARTISTS OR ARTS ORGANISATIONS 



1 a. Number of individual members 




1 b. Number of organisational 
members 




2. Number of artists supported 




3. Support services provided to artists 


a. Advice 

U. 1 d.ll\o/ IcUlUi co 

c. Training 

e. Premises 

f. Residencies arranged 

g. Advocacy 

h. Other, please specify: 


4. Number of enquiries dealt with 




5. Number of talks/lectures 




6. Number of organisations/individuals 
receiving a newsletter 




7. Number of courses 




7a. Number of participants who 
attended courses 




o. NumDer ot nits to weDsite^sj 




9. Number of artists provided with 
premises 




10. Number of residencies arranged 




1 1 . Value of resources levered into the 
sector 
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NOTES FOR THE FINANCIAL FORM 

The Financial Form consolidates information formerly required in financial 
returns to the Arts Council. 
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FINANCIAL FORM 






Expenditure 




Core costs 




1 . Administrative salaries 




2. Rent and rates 




3. Light and heat 




4. Maintenance, security, etc. 




5. Telephone and fax 




6. Printing, postage, stationery 




7. Financial costs 




8. Insurance 




9. Other (please details) 




Programming costs 




10. Artists costs 




1 1 . Volunteer expenses 




12. Publicity, marketing 




13. Catering merchandising 




14. Transportation 




15. Travel, accommodation, 




subsistence 




16. Equipment, materials 




17. Other (please detail) 




Total costs 




Income 




Income from grants 




1. Arts Council of Northern Ireland 




(ACNI): revenue 




2. Arts Council of Northern Ireland 




(ACNI): lottery 




3. AnChomhairle Ealaion 




4. Department of Culture, Arts and 
Leisure (DCAL) 




5. Local authority subsidy 




6. European Commission 




7. Other grants 




Earned income 




8. Box office 




9. Domestic touring 




10. International touring 




1 1 . Membership fees 




12. Book sales 




13. Publications 




14. Gallery sales 




1 5. Bar/catering sales 




16. Rent from tenants 




17. Service provision 
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18. Training 




1 9. Other earned income 




Total earned income 




Contributed income 




20. Sponsorship 




21 . Donations 




Total income 




Total capital income 




22. Arts Council of Northern Ireland. 




23. Other capital 




24. Total income 
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NOTES ON THE ATTENDANCE SHEET 

The individual form is necessary for all projects where 90% of the participants 
are over 1 7. A random sample of these individuals will be included in the Arts 
Council's Follow up Survey. 
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ATTENDANCE SHEET FOR PARTICIPATION IN THE FOLLOW UP SURVEY 



Participant's name 


Telephone 


Full post 
code * 
(optional) 


Age 


Gender 


Disability 
(Yes/no) 


Employment 

circumstances 

(code) 


New/repeat 


Sessions: please fill in how much of the 
session the individual attended (all, most 
or some) 


1 


2 


3 


4 


5 


6 


7 


8 


9 


1. 


































2. 


































3. 


































4. 


































5. 


































6. 


































7. 


































8. 


































9. 


































10. 



































NOTES FOR THE ACTIVITY FORM FOR PARTICIPATION 



You should complete enough Activity Forms to cover all of your participative 
work. If your work falls naturally into projects then you should complete one 
form for each project. If your work is more continuous, and is not composed of 
projects, then you need to think about how to define units of activity: these 
could be according to time periods (terms) or structure (partnerships). 

The Activity Form for Participation should be completed by the person who is 
running the session, or by any other member or staff or volunteer who is 
present during the session. The same person could complete the Activity 
Evaluation Form and any quality forms you are using. 

It is important that forms are completed during the session when events are 
fresh, rather than afterwards. You should check the form at the end of the 
session to ensure that it is complete. 

This simple form produces a great deal of interesting information, for example: 

■ The art form representation of your projects. 

■ Number of events. 

■ Number of participants. 

■ Number of participants from socially disadvantaged areas. 

■ Number of events in non arts venues. 

■ Numbers of events targeted at priority groups. 

■ Audience numbers. 

■ Average contact hours per participant. 

■ Staff ratios for sessions. 

■ Average number of sessions per project. 

■ The percentage of your projects that are cross community. 

■ Unit cost per participant. 

■ Your dependency on the Arts Council. 
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ACTIVITY FORM FOR PARTICIPATION 



1 . Name of organisation 




2. Name of project 




3. Art form(s) of project {Please tick all 
that apply) 


a. Music 

b. Drama 

c. Dance 

d. Literature 

e. Visual arts and 
crafts 

f. Film and video 


g. Architecture and 
murals 

h. Traditional arts 

i. Combined arts 
j. Other, please 
specify: 


4. Venue(s) for the 
project 


a. School 

b. College of further or 
higher education 

c. Youth club 

d. Other community 
org. 

e. Library 


f. Museum 

g. Theatre 

h. Gallery 

i. Other arts venue 

j. Other venue, please 
specify: 


5. Post code of venue 




6. Post code of target community 




7. Start and end dates for the delivery 
of the project 


Start: 


End: 


8. Total number of sessions 




y. i otai contact nours per participant 
(number of hours times number of 
sessions) 




1 0. Number of staff present in the 
sessions: 


Present: 


All of the time 


Most of the 
time 


Some of the 
time ! 


1 0a. Number of artists 








1 0b. Number of administrative staff 










1 0c. Any partner staff present 










11. Target number of participants 










1 2. Actual number of participants 










1 3. Age group of participants: 




Number below 1 1 




Number 12-16 




Number 17-24 




Number 25-39 




Number 40-59 




Number 60-79 




Number 80+ 





Annabel Jackson Associates Report to the Arts Council of Northern Ireland 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 

61 



14. Any targeting 


a. 

socially/economically 
deprived groups 

b. Rural groups 

c. Ethnic minorities 

d. Offenders 


e. People with 
disabilities 

f. Victims 

g. None 

h. Other, please 
specify: 


1 5. Any methods to support access 


a. Outreach 

c. Childcare 

d. Affordability 

e. Help with 
transportation 

f. Publicity available in 
different languages 

g. Publicity available in 
different formats 


h. Accessible venue 

i 1 Iqp nf Qinn 

1 ■ UOC Ul OIL) 1 1 

language/speech to 
text 

j. Support worker 
costs 

k. Sensory interaction 
1. Audio described 
m. None 
n. Other, please 

cnopif \/ ■ 


16. Any capacity building techniques 
used 


a. Active participation 

b. Skill building 

c. Basing work on 
participants 

irloac/icci ioc 

lUtJclo/ looUCo 

d. Sense of place 

e. Building group 

jrlpntitv 


f. Giving the group 
ownership of the 
product 

g. Leaving a legacy of 

oni linmont for fiitiirn 
CLjUipiiiciii ivji lULUit; j 

work 

h. Participants 

naccinn nn thoir ckillQ 

pCtOOll ILJ VJI 1 11 ICII Ol\IMO ! 

to the community 


17. Religion of participants 


a. All Catholic 

b. All Protestant 

c. Mainly Catholic 

d. Mainly Protestant 


e. Cross community 

f. Do not know 

g. Other religion, 
please specify: 


1 8. Charge paid by participants 


a. Free 

b. Token charge to 
participants 


c. Full charge 

d. Other, please 
specify: 


1 9. Acknowledgement of participation 


a. No 

b. Yes, certificate 

c. Yes, qualification 


20. End product 


a. No 

b. Yes, a performance 

c. Yes, an exhibition of art work or craft work 

d. Yes, a film or video 

e. Yes, participants attended a show 


20a. If there was a performance or 
exhibition, please estimate the 
audience numbers 




21 . Total cost of project 




22. Estimate of ACNI core funding 

attributable to this project (the number of 
workshops or events for this project divided by 
your total number of workshops or events 
times total core funding) 




23. ACNI project funding 
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ACTIVITY FORM FOR PRODUCTIONS/CONCERTS 



1 . Name of organisation 




2. Name of production/concert 




3. Were the performers paid? 


a. All 
h Most 

kJ . 1 VI WO I 

c. Some 

d. None 


4. Number of professional artists 
employed 




5. Origin of production 


a. Own production 

b. Co-production 

c. Tour from Nl company 

d. Tour from company based outside Nl 


6. Where else touring 


a. Within Nl 

b. Rest of the UK 

c. Republic of Ireland 

d. Outside the UK 


7. Was the production/script a new 
work? 


a. Yes 

b. No 
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ACTIVITY FORM FOR PERFORMANCES/CONCERTS (ONE FORM PER 
VENUE) 



1 . Name of organisation 




2. Name of production 




3. Was the venue at your home base? 




3a. If not, what was the post code of 
the venue? 




4. Type of venue 


a. Theatre 

b. Concert hall 

c. Gallery 

d. Other arts venue 

e. School 

f. College of further or 
higher education 

g. Youth club 

h. Other community org. 
i Librarv 

j. Museum 

k. Other venue, please specify: 


5. Targeting 


a. Children/young people 

b. Socially/economically 
deprived groups 

c. Rural groups 

d. Ethnic minorities 

e. Offenders 

f. People with disabilities 

g. Victims 

h. The general population 

i. None 

i Other nlease snecifv 


6. Any methods to support access 


a. Outreach 

K f^rnim tarnotinn 
u. \j\ uu|j Let i y ol.ii ly 

c. Pricing 

d. Help with transportation 

e. Publicity available in different languages 

f. Publicity available in different formats 

g. Accessible venue 

h. Use of sign language 

i. Support worker costs 
j. Sensory interaction 
k. Audio described 

1. None 

m. Other, please specify: 


7. Date of first performance 




8. Date of last performance 




9. Total number of performances at 
this venue 
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10. Seating capacity 




1 1 . Percentage of available tickets 
sold 




12. Total number of tickets sold 




1 3. Number of full price tickets sold 




14 Numbpr of di<?(":ountpd or frpp 
tickets distributed 




1 5. Number of concessionary tickets 
sold 




16. Minimum ticket price 




17. Maximum ticket price 




18. Total audience numbers 
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ACTIVITY FORM FOR EXHIBITIONS 



1 . Name of exhibition 

2. Post code of venue 




3. Date started 




4. Date ended 




5. Exhibitions days 




6. Type of venue 


c. Gallery 

a. Theatre 

b. Concert hall 

d. Other arts venue 

e. School 

f. College of further or 
higher education 

g. Youth club 

h. Other community org. 

i. Library 

j. Museum 

k. Other venue, please specify: 


7. Targeting 


a. Children/young people 

b. Socially/economically 
deprived groups 

c. Rural groups 

d. Ethnic minorities 

e. Offenders 

f. People with disabilities 

g. Victims 

h. The general population 

i. None 

j. Other, please specify: 


8. Any methods to support access 


a. Outreach 

b. Group targeting 

c. Childcare 

d. Affordability 

e. Help with transportation 

f. Publicity available in different languages 

g. Publicity available in different formats 

h. Accessible venue 

i. Use of sign language 
j. Support worker costs 
k. Sensory interaction 

I. Audio described 
m. None 

n. Other, please specify: 


9. Charging paid by visitors 


a. Free 

b. Token charge to visitors 

c. Full charge 

d. Other, please specify: 
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I u. ^riyiii ui pcriuii iidrioc 


a Own nrodurtion 

b. Co-production 

c. Tour from Nl company 

d. Tour from company based outside Nl 


1 1 . Where else touring 


a. Within Nl 

b. Rest of the UK 

c. Republic of Ireland 

d. Outside the UK 


1 2. Number of artists exhibited 




13. Total number of visitors 




14. How visitor numbers are produced 


a. Counter 

b. Estimate 

c. Sample count , 
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ACTIVITY FORM FOR PUBLICATIONS 



1 . Name of organisation 




2. Name of publication 




3. Type of publication 


a. Book 

b. Periodical 

c. Exhibition catalogues 

d. CD Rom 

e. Newsletter 

f. Other, please specify: 


4. Age of publication 


a. New title 

b. Backlist 


5. Language 


a. English 

b. Irish 

c. Ulster Scots 

d. Other, please specify: 


vj . AAl iy 1 1 lull IVJvJo L\J II IOI CCloC dOUCoj 


o. Large print 
p. Talking books 
m. None 

n. Other, please specify: 


7. Number of copies of the publication 
produced 




8. Number of copies of the publication 
sold 




9. Numbers of copies of the 
publication distributed free 





Annabel Jackson Associates Report to the Arts Council of Northern Ireland 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 



68 



APPENDIX THREE: OPTIONAL QUESTIONNAIRES 

Optional questionnaires are provided: 

■ To help you collect information for the obligatory questionnaires. 

■ To save you time in writing questionnaires for other areas of interest to your 
organisation. 

■ To encourage you to think about the way that evaluation can help your 
organisation to learn from experience. 

Recording activity 

NOTES FOR THE ENROLMENT FORM 

Keep track of activity as you go along. It is much more difficult and time- 
consuming to try and reconstruct a description of your work after the event. 
Record what happened, not what could have happened, might have happened, 

should have happened or will happen. 

This information is simple but very useful. It is surprisingly common for arts 
organisations not to know how many participants they have or had. This puts 
them at a great disadvantage in raising funds: if you have done the work, you 
should take the credit for it. 

The main point in measuring the outputs of voluntary and community sector arts 
organisations is to demonstrate the depth rather than the breadth of participation. 
You are unlikely to have large numbers of participants, but you could have 
intensive, detailed participation which you will want to communicate in your 
evaluation. This is not difficult to do. 

You should collect basic information on all project participants. The only 
exception would be for drop-in projects where participation in the project is 
relatively brief and inconsistent. However, wherever you can, please collect this 
information as it will help you to understand and explain your work. 

You might already have enrolment forms, in which case you might like to amend 
your existing forms to include any information on the toolkit's form that you do not 
currently have. 

Enrolment Forms do not have to be completed at the first session of a workshop. 
Some arts organisations will prefer to collect some of this information at the 
second contact when some level of trust has been established and membership 
is more stable. 

This form is needed to provide the contact information for the follow up telephone 
survey. 

The first part of the form can be filled in by the participant. The second part of the 
form can be filled in by the project facilitator or leader or by another individual 
involved in the project such as a teacher. 
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SAMPLE ENROLMENT FORM 



\ . Y our ridiiiG 




2. Address 




^ Pnct pnrlp 
o. nuoL ouuc 




4. Telephone number: home 




5. Telephone number: mobile 




6. Your age 


a. 1 7-24 

b. 25-39 

c. 40-59 


d. 60-79 

e. 80+ 


I . uenuer 


a. Male 


b. Female 


o. vvnat are your employment 
circumstances? (Please tick the 
main one that applies) 


a. Employed full 
time 

b. Employed part 
time 

c. Self employed 
d. 

Volunteer/unpaid 

worker 

e. At ho me 


f. Unemployed 

g. Student 

h. Retired 

i. Other, please 
specify: 


9. Are you a disabled person? 


Yes 


No 


10. Do you use: 


a. Visual aids or alternative formats to 
print i.e. Braille, large print or audio 
tape 

b. Communication support or 
equipment i.e. sign language 
interpreter, speech-to-text, loop etc. 

c. Equipment to aid mobility i.e. wheel 
chair, walking frame etc. 

d. A support worker or personal 
assistance 

e. Other support, please specify: 


1 1 . Have you been involved in 
one of this organisation's arts 
projects before? 


Yes 


No 



For later recording: sessions attended (please tick) 



1 


2 


3 


4 


5 


6 


7 


8 


9 


10 
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NOTES ON THE ATTENDANCE SHEET 

There are two forms of the attendance sheet: for individuals or groups. The 
individual form is necessary for all projects where 90% of the participants are 
over 17. A random sample of these individuals will be included in the Arts 
Council's Follow up Survey. 

Collecting information at the individual level is more accurate and easy to keep 
track of. However, your participants might come to you through a group (e.g. 
school or youth club), in which case you might not know individual details. In this 
case, you might prefer to keep records for the Obligatory Activity Form for 
Participation through a group attendance s heet. This can be completed by your 
artists or facilitators or by the teachers or workers from your partner agency. 
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GROUP ATTENDANCE SHEET 



Name of school/group 


Telephone 


Full post 
code of 
school 


Age 
range 


Gender 


Number of 
participants 
with a 
disability 


Total number Total number 
of participants of sessions 


1. 














2. 














3. 














4. 














5. 
6. 














7. 














8. 














9. 














10. 















* Put ROI for Republic. 
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Recording quality 

NOTES FOR THE QUALITY FORM 

Quality forms are internal forms to help structure your own reflection and 
learning. Select a sample of your sessions for observation, for example, 5-10% of 
sessions. If you are using observation for evaluation purposes then the sessions 
should be selected randomly (for example, every tenth session that you run). 
The same person could complete the Quality Form and the Activity Evaluation 
Form. Be sure to record especially positive or vivid comments verbatim so that 
they can be used (anonymously) in evaluation reports or annual reports. 

EXAMPLE OF ISSUES THAT COULD BE INCLUDED IN A QUALITY FORM 



1 . Name of project 




2. Name of interviewer 




3. Planning 


Is the session well planned? 


4. Resourcing 


Is the funding adequate to the task? 


5. Partnerships 


Are partner agencies present? Do they appear to understand 
and support the project? Have they carried out any 
preparation that was expected? 


6. Venue 


Is the venue easy to find, accessible, the right size and shape 
(if relevant), secure and fitfor purpose? Does the project 
have a clear allocation of space - not shared or double 
booked? 


7. Delivery 


Is the session at a sensible time and of a sensible length? 


8. The participants 


Are there enough'too many participants? Do participants form 
a coherent group? 


9. Engagement of 
participants 


Do participants turn up on time and stay for the duration of 
the session? Do they seem interested and actively involved? 


10. Facilitator 


Does the facilitator appear to have the right technical skills? 
Do they have an empathy with the participants? 


1 1. Technology 


Are there any technical strengths or weaknesses: presence 
or absence of a particular piece of equipment whether for 
sound, lighting, IT or other? 


12. Originality 


Is the work new?This is one of the facets of artistic quality. 


13.Ambition 


Is the work too ambitious or not ambitious enough? 


14. Magic 


Are there examples of individual inspiration? Magic is another 
facet of artistic quality. 


1 5. Relevance to the 
audience/participants 


Is the work right for the ability of the participants? Does the 
facilitator adapt the session to the interests or concerns of 
participants? 


16. Other, please specify: 




1 7. What are the main 
improvements that you think 
we should make to this 
project? 





Annabel Jackson Associates Report to the Arts Council of Northern Ireland November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 



73 



SAMPLE QUALITY FORM 



i . iNcuiie ot project 




^. Name ot inierviewer 






oirengins 


weaknesses 


3. Planning 






4. Resourcing 






5. Partnerships 






6. Venue 






7. Delivery 






8. The participants 






9. Engagement of participants 






10. Facilitator 






11. Technology 






12. Originality 






13.Ambition 






14 Manir 

i i . i v i ciy i w 






1 5. Relevance to the 
audience/participants 






16. Other, please specify: 






1 7. What are the main 
improvements that you think we 
should make to this project? 
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Example from Creative Youth Partnerships, could be used for youth 
arts projects 

NOTES FOR THE OBSERVATION QUESTIONNAIRE 

You could consider obtaining feedback from your staff and partners involved in 
the project. 

Questions should be as precise as possible rather than just asking for strengths 
and weaknesses of the project. If you are precise this will save your interviewees 
time in replying and produce better quality data. 
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OBSERVATION QUESTIONNAIRE FOR YOUTH ARTS 



1 . Name of observer 




2. Receiving organisation 




3. Name of project 




4. Name of artist(s) 




5. Date of observation 




6. Number of children/young people 
present 




7. Was the teacher/youth leader 
present? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


8. Did the teacher/youth leader take 
part? 


a. All the time 

b. Most of the time 

c. A small amount of time 

nl Mn+ n + nil 

d. Not at all 


9. Did the artist(s) and the teacher/youth 
leader work well together? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


"lO.Was the artist well prepared for the 
session? 


a. All the time 

b. Most of the time 

c. A small amount of time 

n 1 M„+ „+ nil 

d. Not at all 


1 1 .Was the activity appropriate for the 
age group present? 


a. All the time 

b. Most of the time 

c. A small amount of time 
a. Not at all 


12. Did the artist set appropriate 
standards for the age group present? 


a. Too high [ 

b. High but realistic 

c. Average 

d. Not stretching 


13.Were the children/young people 
enthused by the session? 


a. All the time 

b. Most of the time 

c. A small amount of time 

ri Mnt nil 

0. not ai an 


14. Did all members of the group 
participate? 


a. All the time 

b. Most of the time 

c. A small amount of time 
d Nnt at all 

u. INUl Ql all 


15.Were the children/young people on 
task during the session? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


16. Did the session encourage the 
children/young people to think 
creatively? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


17. Did the artist(s) use the time 
allocated well? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 
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18.Were the appropriate health and 
safety regulations followed? 


a. Evidence of thought 

b. Not evidence of thought, but no problems 

c. Potential health and safety problem 

d. Health and safety problem 


19.1s there anything you would like to 
see improved? 




20.Are there any lessons for the project? 
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Recording outcomes 

NOTES FOR OUTCOMES QUESTIONNAIRE 

Outcome measurement sounds intangible and difficult but you don't need a high 
level of detail, so it is relatively straightforward. For example, it is enough to ask 
whether participants increased their confidence as a result of the project, not by 
how much their confidence increased. 

The use of social outcomes is an attempt by funders to better match the 
objectives of arts organisations, to move away from a reliance on economic 
impact evaluation. It might sound mechanistic or instrumental, however, the 
voluntary and community arts do have strong effects on people. We hope that 
recording what already occurs can show the sector in a stronger light as well as 
opening some interesting discussions within the sector. 

You might like to change the title of the questionnaire, for example to "Your 
Views". 

Think about whether you can apply this questionnaire as it is in, or if you need to 
shorten it. Participants should be interviewed at the end of the project. Don't just 
interview successes. Include people who dropped out part of the way through the 
project. If you explain that this is part of your review of the project to make the it 
better then most participants will be willing to be interviewed. 

Apply questionnaires to a random sample of participants. It is extremely 
important that the sample is random because otherwise the results will not be 
valid. In practice, it is probably easier to select, for example, every tenth project, 
rather than every tenth participant. However, you must select projects randomly, 
not just the easy or positive projects. The Outcomes Survey is not obligatory, it is 
partly intended to be of value to your organisation and so you need the results to 
be a fair reflection of the cross-section of your work. More information on 
sampling is given in Appendix One. 



The number of outcome questionnaires you collect should be broadly in 
proportion to the scale of your work, subject to a minimum number of 30-47: 



Number of participants across the 
organisation 


Percentage of participants to be 
interviewed 


Under 50 


95-100% 


50-99 


50% 


100-299 


25% 


300+ 


10% 



Analyse the survey by calculating the percentage of respondents/responses for 
each answer. 



Annabel Jackson Associates Report to the Arts Council of Northern Ireland 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 



78 



OUTCOMES QUESTIONNAIRE 



1 . Name of project 




2. Name of organisation 




3. Have you ever taken part in an arts 
project before? 


Yes 


No 


4. Have you been involved in one of 
this organisation's arts projects before? 


Yes 


No 


o. nuuyiiiy iiuw many bobbiuiib uiu yuu 
attend in the projecl/term? 




6. Did you have problems attending? 


Yes, why? 

a. You were busy 

b. The times were inconvenient 

c. The locations were inconvenient 

d. You needed help with childcare 

e. Other, please specify: 


No 


7. Did you enjoy being involved in the 
project? 


a. Yes, loved it c. Neutral d. Didn't enjoy 

b. Yes, liked it it 

e. Disliked it 



What effect, if any, has the project had on you? Are you: 




1 


2 


3 


4 


5 






©© 


© 


© 


© 


©© 




8. More confident? 












Less confident? 


9. More ambitious for 
yourself? 












Less ambitious for 
yourself? 


10. More creative? 












Less creative? 


11. More able to express 
yourself? 












Less able to express 
yourself? 


12. Healthier? 












Less healthy? 


13. Happier? 












Less happy? 


14. More optimistic that 
individuals can make a 
difference? 












Less optimistic that 
individuals can make a 
difference? 


15. More likely to trust 
people? 












Less likely to trust 
people? 


16. More part of society? 












Less part of society? ! 


17. More interested in 
attending arts events? 












Less interested in 
attending arts events? 


18. More interested in 
education or training? 












Less interested in 
education or training? 


19. More interested in 
attending community 
events? 












Less interested in 
attending community 
events? 
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20. Has the project brought you into 
contact with people who have a 
different religion to you? 


Yes 


No 


Unsure/not 
applicable 


21. Has the project helped you to 
understand people who are different to 
you? 


Yes 


No 


Unsure/not 
applicable 


22. Have you gained skills in any of 
inese areas as a resun ot trie project r 


a. Creative arts skills 

b. Technical arts skills 

c. Performing skills 

d. People skills 


e. Project 
management 

f. Project 
administration 

g. Using computers 

h. Communication 
skills 

i. No skills gained 
j. Other, please 
specify: 


22a. If yes, have any of these skills 
helped you: 


a. In home or social life 

b. In school or college 

c. At work 


23. Did you get a qualification as a 
result of taking part in the project? 


Yes: which? 


No 


24. Is there anything else that you 
gained from being involved in this 
project? 


Yes: what? 


No 




25. What did you enjoy most about the 
project? 




26. What did you enjoy least about the 
project? 




27. Does art matter to you more after 
this project? If yes, why? 


a. Yes: why? 


b. No effect 

c. Art already very 
important 


28. Would you like to be involved in 
more of the organisation's arts 
projects? 


a. Definitely yes 

b. Probably yes 


c. Definitely no 
d Probably no 


29. How can the organisation improve 
its arts projects? 
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30. Your name 




31 . The full post code of your home 




32. Your age 


a. 17-24 

b. 25-39 

c. 40-59 


d. 60-79 

e. 80+ 


33. Gender 


Male 


Female 


34. Are you a disabled person? 


Yes 


No 


35. Do you use: 


a. Visual aids or alternative formats to print i.e. 
Braille, large print or audio tape 

b. Communication support or equipment i.e. 
sign language interpreter, speech-to-text, loop 
etc. 

c. Equipment to aid mobility i.e. wheel chair, 
walking frame etc. 

d. A support worker or personal assistance 

e. Other support, please specify: \ 


36. What are your employment 
circumstances? {Please tick the main 
one that applies) 


a. Employed full time 

b. Employed part time 

c. Self employed 

d. Volunteer/unpaid 
worker 

e. At home 


f. Unemployed 

g. Student 

h. Retired 

i. Other, please specify: 


37. Are you on means tested benefits? 


Yes 


No 



THANK YOU 
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OUTCOMES QUESTIONNAIRE FOR YOUNG PEOPLE (11-16 YEARS) 



1 . Name of project 

2. Name of organisation 




3. Have you ever taken part in an arts 
project before? 


Yes 


No ! 


4. Have you been involved i n one of 
this organisation's arts projects before? 


Yes 


No 


5. Roughly how many sessions did you 
attend in the project/term? 




6. Did you enjoy being involved in the 
project? 


a. Yes, loved c. Neutral d. Didn't 
it enjoy it 

b. Yes, liked it e. Disliked it 



What effect, if any, has the project had on you? Are you: 




1 


2 


3 


4 


5 






©© 


© 


© 


© 


©© 




7. More confident? 












Less confident? 


8. More ambitious for 
yourself? 












Less ambitious for 
yourself? 


9. More creative? 












Less creative? 


10. More able to express 
yourself? 












Less able to express 
yourself? 


1 1 . Happier? 












Less happy? 


12. More likely to trust 
people? 












Less likely to trust 
people? 


1 3. More part of society? 












Less part of society? | 


14. More interested in 
attending arts events? 












Less interested in 
attending arts events? 


15. More interested in 
education or training? 












Less interested in 
education or training? 



1 6. Has the project brought you into 
contact with people who have a 
different religion to you? 


Yes 


No 


Unsure/not 
applicable 


1 7. Has the project helped you to 
understand people who are different to 
you? 


Yes 


No 


Unsure/not 
applicable 


1 8. Did you get a qualification as a 
result of taking part in the project? 


Yes: which? 


No 


1 9. Is there anything else that you 
gained from being involved in this 
project? 


Yes: what? 


No 
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20. What did you enjoy most about the 
project? 




21. What did you enjoy least about the 
project r 

22. Does art matter to you more after 
this project? If yes, why? 


a. Yes: why? 


b. No effect 

c. Art already very 
important 


23. Would you like to be involved in 
more of the organisation's arts 
projects? 


a. Definitely yes 

b. Probably yes 


c. Definitely no 
d Probably no 


24. How can the organisation improve 
its arts projects? 





25. Your name 




26. The area you live in 




27. Your age 




28. Gender 


Male 


Female 


29. Are you a disabled person? 


Yes 


No 


30. Do you use: 


a. Visual aids or alternative formats to print i.e. 
Braille, large print or audio tape 

b. Communication support or equipment i.e. 
sign language interpreter, speech-to-text, loop 
etc. 

c. Equipment to aid mobility i.e. wheel chair, 
walking frame etc. 

d. A support worker or personal assistance 

e. Other support, please specify: 



THANK YOU 
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NOTES FOR ACNI FOLLOW UP QUESTIONNAIRE 

Follow up interviews will be done by ACNI. 

Follow up questionnaires measure the lagged effects of your project: the effects 
that occur sometime after the participants attended the project. They can also be 
used to check whether predicted or intended consequences did occur, for 
example, whether someone who said the project increased their interest in 
learning did actually attend training or education of some kind. 

As with the Outcome Questionnaire you should interview a random sample of 
participants. This does not have to be the sample people as interviewed earlier, 
so long as the two groups are comparable. 

Comment means example, explanation or quote. 



Annabel Jackson Associates Report to the Arts Council of Northern Ireland 



November 2004 



Evaluation Toolkit for the Voluntary and Community Arts 



84 



ACNI FOLLOW UP QUESTIONNAIRE 



1 . Name of project 




2. Name of organisation 




3. Have you ever taken part in an arts 
project before? 


Yes 


No 


Comment: 


4. Have you been involved in one of 
this organisation's arts projects before? 


Yes 


No 


Comment: 


5. Roughly how many sessions did you 
attend in the project/term? 




Comment: 


6. Did you have problems attending? 


Yes, why? 

a. You were busy 

b. The times were inconvenient 

c. The locations were 
inconvenient 

d. You needed help with 
childcare 

e. Other, please specify: 


No 


Comment: 


7. Did you enjoy being involved in the 
project? 

Comment: 


a. Yes, loved c. Neutral 
it 

b. Yes, liked it 


d. Didn't 
enjoy it 

e. Disliked it 



What effect, if any, has the project had on you? Are you: 




1 


2 


3 


4 


5 






©© 


© 


© 


© 


©© 




8. More confident? 












Less confident? 


Comment: 


9. More ambitious for 
yourself? 












Less ambitious for 
yourself? 


Comment: : 
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1 


2 


3 


4 


5 




10. More creative? 












Less creative? 


Comment: 


1 1 . More able to express 
yourself? 












Less able to express 
yourself? 


Comment: 


12. Healthier? 












Less healthy? 


Comment: 


13. Happier? 












Less happy? 


Comment: 

14. More optimistic that 
individuals can make a 
difference? 












Less optimistic that 
individuals can make a 
difference? 


Comment: 


15. More likely to trust 
people? 












Less likely to trust 
people? 


Comment: 


1 6. More part of society? 












Less part of society? 


Comment: 


17. Has the project brought you into 
contact with people who have a 
different religion to you? 


Yes 


No 


Unsure/not 
applicable 


Comment: I 


18. Has the project helped you to 
understand people who are different to 
you? 


Yes 


No 


Unsure/not 
applicable 


Comment: 
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1 9. Have you attended arts events 
since the project? 








Comment: 


20. Have you started some training or a 
college course as a result of taking part 
in the project? 


Yes 


No 


Unsure/not 
applicable 


Comment: 


21. Have you taken part in community 
events since the project? 


Yes 


No 


Unsure/not 
applicable 


Comment: 


22. Have you gained skills in any of 
these areas as a result of the project? 


a. Creative arts skills 

b. Technical arts skills 

c. Performing skills 

d. People skills 


e. Project 
management 

f. Project 
administration 

g. Using computers 

h. Communication 
skills 

i. No skills gained 
j. Other, please 
specify: 


22a. If yes, have any of these skills 
helped you: 


a. In home or social life 

b. In school or college 

c. At work 






23. Did you get a qualification as a 
result of taking part in the project? 


Yes: which? 


No 


24. Is there anything else that you 
gained from being involved in this 
project? 


Yes: what? 


No 



25. What did you enjoy most about the 
project? 




26. What did you enjoy least about the 
project? 




27. Does art matter to you more after 
this project? If yes, why? 


a. Yes: why? 


b. No effect 

c. Art already very 
important 
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28. Would you like to be involved in 
more of the organisation's arts 
projects? 


a. Definitely yes 

b. Probably yes 


c. Definitely no 
d Probably no 


29. How can the organisation improve 
its arts projects? 





30. Interviewee's name 




oi . i ne tun post coae ot your nome 




32. Age 


a. 11-16 

b. 17-24 

c. 25-39 

d. 40-59 


e. 60-79 j 

f. 80+ 


33. Gender 


Male 


Female 


34. Are you a disabled person? 


Yes 


No 


35. Do you use: 


a. Visual aids or alternative formats to print i.e. 
Braille, large print or audio tape 

b. Communication support or equipment i.e. 
sign language interpreter, speech-to-text, loop 
etc. 

c. Equipment to aid mobility i.e. wheel chair, 
walking frame etc. 

d. A support worker or personal assistance 

e. Other support, please specify: 


36. What are your employment 
circumstances? {Please tick the main 
one that applies) 


a. Employed full time 

b. Employed part time 

c. Self employed 

d. Volunteer/unpaid 
worker 

e. At home 


f. Unemployed 

g. Student 

h. Retired 

i. Other, please specify: 


37. Are you on means tested benefits? 


Yes 


No 



THANK YOU 
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NOTES ON YOUTH QUESTIONNAIRES 

These questionnaires were designed for use with Creative Youth Partnerships. 
However, they might be useful for organisations working with young people, 
especially where participants are too young to complete Outcomes Forms 
themselves. 

Generally speaking forms should be completed at the end of the project. 
However, if there is a lag between the workshops and a performance you might 
like to split the questionnaires so that the process questions are asked at the end 
of the workshops, and the outcome questions after the performance. 
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ARTIST'S EVALUATION FORM 



1. 


Contact Information 


1. 


Name of project 


2. 


Date of project 


3. 


Name of artist 



2. Preparing the Project 



4. Were the objectives of the project 
clear? 


a. Yes 

b. No, but it was 

c. No, and it was 


not a problem 
a problem 


5. What information did you get from the 
site visit/planning process? 


a. The age group of children/young people 

b. The abilities of the children/young people 

c. Curriculum links 

d. Contact with the teacher/youth leader 

e. The space and facilities available 
f Other, please specify: 


6. In retrospect, did you get all the 
information you needed during the site 
visit/planning process? 


Yes 


No: what else did you need to know? 


7. Did you feel you had a working 
partnership with the teacher/youth 
leader? 


Yes 
No 


How? 


3. Delivering the Project 


8. Was the Creative Youth Partnerships 
development officer helpful? 


Yes 
No 

No contact 


How? 


9. Did you receive the support you 
needed from the receiving 
organisation? 


Yes 
No 


How? 


10. Did the space available suit what you 
wanted to do? 


Yes 
No 


How? 


1 1 .Did you find the children/young people 
easy to engage? 


Yes 
No 


How? 


12. Did the children/young people 
contribute to the planning of the 
sessions? 


Yes 
No 


How? 


13. Did you encounter any other problems 
in delivering the project? 

14. Were they addressed? 


Yes 
No 

Yes 
No 


What? 
How? 
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4. Your Experiences 



1 5. Did the project contribute to your Yes 
artistic development? 


No 


16. Did the project increase your Yes 
knowledge of the education/youth 
service? 


No 


17. Did you learn about ways to link your Yes 
art form to the curriculum? 


No 


1 8. Did the project highlight gaps in your Yes: how? 
skills or knowledge that Creative Youth 
Partnerships could help you with? 


No 



5. Effects on the Children/Young People 



19. Did the children/young people change by 
the end of the project? 


No 


Yes 


a. If yes, did their confidence increase? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. AII in the group 

e. Difficult to tell 

f. Not relevant, already strong 


b. Did their ability to concentrate increase? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


c. Did their behaviour in the group improve? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


d. Were their attitudes to the arts more positive? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. AII in the group 

e. Difficult to tell 

f. Not relevant, already strong 


e. Were their attitudes to learning more positive? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


f. Other, please specify: 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 
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6. The Achievements of the Project 



20. Do you think the project was a success? 


Yes 

Partly 

No 


How? 


21 .In retrospect, is there anything you would 
have done differently? 




22.1s there anything about Creative Youth 
Partnerships that you think should be 
changed? 


Yes 
No 


What? 



THANK YOU 
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TEACHER'S/YOUTH LEADER'S EVALUATION FORM 

1. Contact Information 

1 . Name of teacher/youth leader 

2. Date of project 

3. Name of artist(s) 

4. Name of project 



2. Before the Project 



5. Were the objectives of the project 
clear? 


a. Yes 

b. No, but it was not a problem 

c. No, and it was a problem 


6. What was your reason for being 
involved? 


a. To give the children/young people a new experience 

b. To develop your skills 

c. To work with a professional artist 

d. To support the curriculum 

e. To release time for other work duties 

f. Other, please specify: 


7. Did the young people contribute to the 
objectives of the project? 


Yes 
No 


How? 



3. The Project 



8. Did you attend the sessions? 


Yes 


No, please go to next section 


9. Was the artist well prepared for the 
sessions? 


a. All the time 

b. Most of the time 

c. A small amountof time 

d. Not at all 


"lO.Was the activity appropriate for the 
age group present? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


1 1 .Were the children/young people 
enthused by the sessions? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


12. Were all members of the group 
engaged? 

13. Were the children/young people on 
task during the sessions? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 

a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


14. Did the sessions encourage the 
children/young people to think 
creatively? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 


15. Did the artist(s) use the time allocated 
well? 


a. All the time 

b. Most of the time 

c. A small amount of time 

d. Not at all 
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16. Did you feel that you had a working 
partnership with the artist(s)? 


Yes 
No 


How? 


4. Your Experiences 


17. Did the project change your view 
about the role of the arts in youth 
development? 


a. Yes, more How? 
positive 

b. Yes, more 
negative 

c. No, already 
very positive 

d. No, other 


18. Did the project change the status of 
the arts within your organisation? 


a. Yes, more How? 
positive 

b. Yes, more 
negative 

c. No, already 
very positive 

d. No, other 


19.1s there anything in your own 
professional practice that you would 
do differently because of the 
experience of the project? 


Yes What? 
No 


20. Do you feel more confident integrating 
arts activity into your ongoing work? 


Yes How? 
No 


21 .Would you use the same artist again? 


Yes Why? 
No 


22.Would you like to take part in future 
arts projects? 


Yes Why? j 
No 


5. Effects on the Children/Young People 


23. Did the children/young people change 
by the end of the project? 


No 


Yes 


a. If yes, did their confidence increase? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


b. Did their ability to concentrate increase? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


c. Did their behaviour in the group improve? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


d. Were their attitudes to the arts more positive? 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell i 
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f. Not relevant, already strong [ 


e. Were their attitudes to learning more positive? 


a. None in the group 

h A fpw in thp nrmin 

u. n ICVV III uic y i uup 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 


f. Other, please specify: 


a. None in the group 

b. A few in the group 

c. Most in the group 

d. All in the group 

e. Difficult to tell 

f. Not relevant, already strong 



6. Your Evaluation of the Project 



24. Do you think the project was a Yes 
success? ^ a rtly 


How? 


25. How could it have been improved? 


26.1s there anything about our overall Yes 
programme of work that you think No 
should be changed? 


What? 



THANK YOU 
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ORGANISATIONAL HEAD'S EVALUATION FORM 

1. Contact Information 

1 . Name 

2. Date of project 

3. Name of project 



2. Before the Project 



4. How much money did your 

organisation spend on professional 
artists last year? 


£ 


5. Is your use of professional artists 
changing? 


a. Increasing 

b. Staying the same 

c. Decreasing 


6. How easy is it to find out what artists' 
programmes are available? 


a. Easy 

b. OK 

c. Difficult 


Comment: 


3. Planning the Project 


7. Did the project fit in with your overall 
organisational objectives? 


a. Yes, strongly 

b. Yes, partly 

c. No 


8. What were your objectives for the 
project? 


a. To give the children/young people a new experience 

b. To develop teachers'/youth leaders' skills 

c. To work with a professional artist 

d. To support the curriculum 

e. Other, please specify: 


9. How easy was it to motivate 

teachers/youth leaders to take part? 


a. Easy 

b. OK 

c. Difficult 


Comment: 


10. How easy was it to schedule the 
project into the curriculum? 


a. Easy 

b. OK 

c. Difficult 


Comment: 


1 1 .How easy was it find the money for 
the project? 


a. Easy 

b. OK 

c. Difficult 


Comment: 



4. Your Evaluation of the Project 



12. Did you think the project had enduring Yes 
effects on the teachers/youth leaders? No 


If yes, did it: 

a. Show them new ways of teaching 

b. Increase their interest in the arts 

c. Other, please specify: 


13. Did you think the project had enduring Yes 
effects on the pupils? No 


If yes, did it: 

a. Increase their confidence 

b. Increase their interest in learning 

c. Increase their interest in the arts 

d. Other, please specify: 
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14. Do you think the project was a a - Yes 

o b. Partly 

success? cNo 


Please explain: 


1 5.Was the project value for money? a - ^ 

c. No 


Why? 


16. How could it have been improved? 


17.1s there anything about our overall Yes 
programme of work that you think No 
should be changed? 


What? 



THANK YOU 
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APPENDIX FOUR: DATA PROTECTION FORM 
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